# realestatebr 1.0.1

CRAN resubmission. Version 1.0.0 was archived on 2026-05-28 because the
package and its transitive `piggyback`/`gh` dependencies wrote to
`~/.cache/realestatebr/` and `~/.cache/R/gh/`, violating CRAN's policy on
home-filespace writes. This release removes this: the caching
architecture is fully overhauled.

## Caching architecture

The user-level disk cache has been removed. The package no longer writes
to `~/.cache/realestatebr/` (or any other location outside the R session's
temporary directory).

* `get_dataset()` is now two-tier: it tries the package's GitHub release
  asset first and falls back to a fresh download from the original source.
  The `source` argument no longer accepts `"cache"`; valid values are
  `"auto"`, `"github"`, and `"fresh"`. The `max_age` argument has been
  removed (cache freshness is now managed by the weekly CI/CD release
  pipeline).
* Repeated `get_dataset()` calls within one R session are served from a
  package-private in-memory environment. Use the new
  `clear_session_cache()` function to drop it.
* The exported helpers `clear_user_cache()`, `check_cache_status()`, and
  `update_cache_from_github()` have been removed.
* Internal dataset functions (`get_abecip_indicators()`, `get_secovi()`,
  `get_rppi_*()`, etc.) no longer accept a `cached` argument. They always
  download from the original source; use `get_dataset(source = "github")`
  for the pre-processed asset.
* `piggyback` (formerly Suggests) and `rappdirs` (formerly Imports) have
  been dropped. GitHub release assets are now fetched directly via
  `httr::GET()` against the public release-asset URL, avoiding the
  transitive `gh::gh()` cache writes that also violated CRAN policy.

## Bug fixes

* `attach_dataset_metadata()` now accepts `"bundled"` as a valid `source`
  value, fixing a hard error when `get_dataset("abecip", "cgi")` was
  called with `source = "fresh"`. The CGI table loads from a bundled
  `inst/extdata` file and was previously tagged with the rejected value
  `"cache"`.

## Internal

* Removed the unused `data-raw/publish-cache.R` script, which still
  imported the dropped `piggyback` dependency. The weekly workflow has
  uploaded `data-raw/cache_output/` via `gh release upload` since the
  caching refactor.

# realestatebr 1.0.0

## Breaking changes

* `nre_ire` has been removed. It required fully manual updates and had no
  automated pipeline support; it will be reconsidered for a future release.
* `property_records` has been removed because the upstream data source is no
  longer available.
* `cbic` has been removed. The upstream CBIC portal migrated to a
  restricted-access platform. The five cement tables will be rebuilt from
  IBGE open data in a future release.
* `itbi_summary` and the internal ITBI helpers (`get_itbi`, `get_itbi_bhe`)
  have been removed. They were incomplete (single-municipality coverage) and
  are deferred to a future version.

## Changes to `bcb_series`

* `get_dataset("bcb_series")` now returns only four columns: `date`,
  `code_bcb`, `name_simplified`, and `value`. Full metadata is available
  via `bcb_metadata`.
* The `table` argument now accepts a hierarchy level: `"core"` (default),
  `"primary"`, `"secondary"`, `"tertiary"`, or `"full"`. The levels are
  cumulative — `"primary"` includes all `"core"` series plus key macro
  indicators such as SELIC, IPCA, and INCC. Previously the argument
  accepted BCB category names (`"credit"`, `"price"`, etc.).
* `bcb_metadata` gains a `hierarchy` column (integer 1–4) that records the
  relevance tier assigned to each series.

## Changes to `rppi`

* `get_dataset("rppi", table = "all")` now returns two additional columns:
  `transaction_type` (`"sale"` or `"rent"`) and `source` (the index name,
  e.g., `"IGMI-R"`, `"IVG-R"`, `"FipeZap"`). Previously the stacked table
  had no way to distinguish transaction type from index source.
* Fixed a sheet-encoding mismatch in the FipeZAP XLSX download that caused
  `Sheets not found` errors after FIPE added a new summary sheet. Sheet
  selection now uses numeric indices to avoid Latin-1/UTF-8 mismatches in
  accented sheet names.
* Fixed `get_range()` failing with a malformed cell range when `tidyxl`
  received a sheet name with a mismatched encoding attribute. The function
  now derives the sheet lookup key directly from the cells returned by
  `tidyxl` rather than from the user-supplied string.

## Bug fixes

* Restored `fetch_fgv_local()`, which was inadvertently dropped during
  earlier refactoring but is still called by the `targets` pipeline to
  process the manually-maintained FGV IBRE CSV export.

## Internal changes

* Dataset download functions now follow a `download_*` / `clean_*` naming
  convention: `download_*` functions return a file path; `clean_*` functions
  parse and tidy that path into a tibble.
* Replaced bare `tryCatch` with `rlang::try_fetch` throughout, using
  `parent = cnd` to preserve the original error chain.
* All dataset functions now call `validate_dataset_params()`,
  `attach_dataset_metadata()`, and `validate_dataset()` from
  `R/helpers_dataset.R` and `R/helpers_download.R` instead of
  re-implementing these patterns inline.

## CI / pipeline

* Fixed stale target names in `.github/workflows/update_data_weekly.yml`.
  The workflow was silently skipping `abecip` and `abrainc` targets on every
  automated run; those entries now use the current granular target names
  (`abecip_sbpe_data`, `abecip_units_data`, `abrainc_indicator_data`, etc.).
* Added `fgv_ibre_file` to the weekly and `all` target groups so the FGV
  file-change target is included in scheduled runs.

## Documentation

* Updated the `bcb_series` table reference in `vignettes/getting-started.Rmd`
  to use the new hierarchy levels (`"core"`, `"primary"`, `"secondary"`,
  `"tertiary"`, `"full"`) instead of the removed category names.
* Updated the `rppi_bis` table listing to include `detailed_annual` and
  `detailed_halfyearly`, which were previously omitted.

# realestatebr 0.6.1

## CRAN Submission Fixes

- Upgraded HTTP URLs to HTTPS in `get_secovi.R`
- Added `URL` field to DESCRIPTION
- Fixed incomplete `@source` tag for `dim_city` dataset documentation
- Fixed typo in `b3_real_estate` documentation ("mian" -> "main")
- Removed local `skip_on_cran()` definition that shadowed `testthat`
- Updated stale description for CBIC dataset in registry

# realestatebr 0.6.0 (2025-11-09)

## Cache Freshness Detection (Phase 2)

**Version 0.6.0 introduces an intelligent cache freshness detection system with relaxed defaults to avoid annoying users with unnecessary warnings.**

### New Features

#### Cache Age Tracking
- **`get_cache_age()`**: Returns cache age in days for any dataset
- **`is_cache_stale()`**: Checks if cache exceeds recommended freshness thresholds
- **`check_cache_status()`**: Diagnostic function showing status of all cached datasets

#### Relaxed Warning Thresholds
Cache warnings only appear when data is **significantly stale** (exceeds 2x the update frequency):
- Weekly datasets: warn after 14 days (not 7)
- Monthly datasets: warn after 60 days (not 30)
- Manual datasets: never warn

#### Advanced Cache Control
- **`max_age` parameter** in `get_dataset()`: Force fresh download if cache exceeds specified age
- Useful for users who need very recent data
- Most users won't need this - relaxed defaults handle typical use cases

#### Registry Enhancements
All datasets in `inst/extdata/datasets.yaml` now include:
- `update_schedule`: "weekly", "monthly", or "manual"
- `warn_after_days`: Custom threshold for staleness warnings (NULL for manual datasets)

### Examples

```r
# Check status of all cached datasets
check_cache_status()

# Get age of specific dataset
get_cache_age("bcb_series")

# Check if dataset is stale (uses relaxed defaults from registry)
is_cache_stale("bcb_series")

# Advanced: Force very fresh data (< 1 day old)
get_dataset("bcb_series", max_age = 1)

# Advanced: Only use cache if less than 3 days old
get_dataset("rppi", table = "sale", max_age = 3)
```

### Tests
- Added 8 comprehensive tests for cache freshness system
- All tests passing (41 total cache tests)
- Validated relaxed thresholds and max_age behavior

## Code Simplification: Logic Consolidation (Phase 3)

### Generic Helper Functions
**Version 0.6.0 introduces 7 generic helper functions that consolidate 890 lines of repetitive code patterns across dataset functions.**

#### What Changed
- **Created**: R/helpers-dataset.R with 6 new helper functions (430 lines)
- **Refactored**: 5 core files using new helpers (417 lines removed, 15.4% reduction)
- **Added**: apply_table_filtering() in R/get-dataset.R (95 lines, eliminates 156 lines of duplication)
- **Added**: 52 comprehensive tests for all helper functions
- **Improved**: Consistent error messages and metadata across all datasets

#### File-by-File Results

| File | Before | After | Lines Saved | % Reduction |
|------|--------|-------|-------------|-------------|
| get_abecip_indicators.R | 551 | 431 | 120 | 21.8% |
| get_abrainc_indicators.R | 544 | 445 | 99 | 18.2% |
| get_secovi.R | 438 | 356 | 82 | 18.7% |
| get_bcb_series.R | 334 | 278 | 56 | 16.8% |
| get-dataset.R | 833 | 773 | 60 | 7.2% |
| **TOTAL** | **2,700** | **2,283** | **417** | **15.4%** |

#### New Helper Functions

1. **validate_dataset_params()** (R/helpers-dataset.R)
   - Consolidated input validation for table, cached, quiet, max_retries parameters
   - Ensures consistent error messages across all datasets
   - Saves ~28 lines per file

2. **handle_dataset_cache()** (R/helpers-dataset.R)
   - Unified cache loading with fallback strategies
   - Consistent error handling and user messages
   - Saves ~35-50 lines per file

3. **attach_dataset_metadata()** (R/helpers-dataset.R)
   - Standardized metadata attachment (source, download_time, download_info)
   - Flexible extra_info parameter for dataset-specific metadata
   - Saves ~8-16 lines per file

4. **validate_dataset()** (R/helpers-dataset.R)
   - Generic data validation (rows, columns, dates)
   - Configurable validation rules with detailed error messages
   - Saves ~44 lines per file

5. **validate_excel_file()** (R/helpers-dataset.R)
   - Excel file validation (size, expected sheets)
   - Used by abrainc and abecip functions
   - Prevents silent failures

6. **download_with_retry()** (R/rppi-helpers.R - REUSED)
   - Found existing implementation, avoided duplication
   - Saves ~46-86 lines per file that would have been duplicated

7. **apply_table_filtering()** (R/get-dataset.R)
   - Centralizes all table/category filtering logic
   - Supports property_records, SECOVI, BCB Real Estate, BCB Series
   - Eliminates 156 lines of duplication between cache functions

#### Files Changed
- **New**: R/helpers-dataset.R (430 lines, 6 helpers, 52 tests)
- **Updated**: R/get_abecip_indicators.R (21.8% reduction)
- **Updated**: R/get_abrainc_indicators.R (18.2% reduction)
- **Updated**: R/get_secovi.R (18.7% reduction)
- **Updated**: R/get_bcb_series.R (16.8% reduction)
- **Updated**: R/get-dataset.R (7.2% reduction, added apply_table_filtering())

#### Rationale
- **Simplification**: Reduce codebase complexity and maintenance burden
- **Consistency**: Ensure uniform behavior across all dataset functions
- **DRY**: Follow "Don't Repeat Yourself" principle
- **Testing**: Well-tested helpers prevent regressions

See `.claude/phase3_completion_summary.md` for complete details.

---

## BREAKING CHANGES: API Simplification (Phase 2)

### Removed Deprecated Function Exports
**Version 0.6.0 removes 8 deprecated functions from the public API. These functions are now internal-only.**

#### What Changed
**Removed from NAMESPACE**: 8 deprecated functions no longer exported:
  - `get_abecip_indicators()`
  - `get_abrainc_indicators()`
  - `get_bcb_realestate()`
  - `get_bcb_series()`
  - `get_fgv_ibre()`
  - `get_nre_ire()`
  - `get_rppi_bis()`
  - `get_secovi()`

#### Impact
- **Users must migrate**: These functions can no longer be called directly
- **get_dataset() is the only supported API**: All data access must go through `get_dataset()`
- **Cleaner public interface**: Package exports only essential user-facing functions

#### Migration
These functions were deprecated in v0.4.0 (18+ months ago). Users must now use `get_dataset()`:

```r
# Old way (NO LONGER WORKS):
data <- get_secovi()
data <- get_bcb_series(table = "price")
data <- get_abecip_indicators(table = "sbpe")

# New way (REQUIRED):
data <- get_dataset("secovi")
data <- get_dataset("bcb_series", "price")
data <- get_dataset("abecip", "sbpe")
```

#### Rationale
- **Simpler API**: One function (`get_dataset()`) instead of 15+
- **Pre-1.0.0 flexibility**: Breaking changes acceptable before stable release

### Code Clarity Improvements

#### Renamed Confusing "Legacy" Terminology
- **Renamed**: `get_from_legacy_function()` → `get_from_internal_function()`
- **Rationale**: These functions call internal worker functions, not "legacy" code
- **Impact**: Internal only - no user-facing changes

**Files changed**: `R/get-dataset.R`

### CBIC Code Simplification and Table Availability

#### Code Reduction (~223 lines, 11% smaller)
- **Removed**: 100+ lines of commented-out old implementation code
- **Removed**: 4 unused helper functions (124 lines total):
  - `suppress_external_warnings()` - Never called
  - `explore_cbic_structure()` - Only in examples
  - `get_cbic_files()` - Only in examples
  - `get_cbic_materials()` - Only in examples
- **Removed**: Unnecessary metadata attributes from `get_cbic_steel()` and `get_cbic_pim()`:
  - `attr(result, "source")`
  - `attr(result, "download_time")`
  - `attr(result, "download_info")`
  - Associated tryCatch blocks and cli_user messages (~69 lines)

#### Table Availability Fixed
- **Unblocked**: `steel_prices` and `pim` tables now accessible
- **Blocked**: Only `steel_production` remains blocked (has data quality issues)
- **Updated**: Error messages now accurately reflect v0.6.0 status
- **Available tables**:
  - Cement: cement_monthly_consumption, cement_annual_consumption, cement_production_exports, cement_monthly_production, cement_cub_prices
  - Steel: steel_prices
  - PIM: pim, pim_production_index

#### Rationale
- **Simplification**: Removed dead code and unused functions for better maintainability
- **Accuracy**: Updated table availability to reflect actual working status
- **User experience**: Clear error messages guide users to available tables

**Files changed**: `R/get_cbic.R`

---

## BREAKING CHANGES: Documentation Simplification (Phase 1)

### Removed Examples from Deprecated Functions
**Version 0.6.0 removes usage examples from deprecated legacy functions to simplify the codebase. Since we are pre-1.0.0, this is an acceptable breaking change.**

#### What Changed
- **Removed**: All `@examples` blocks from 8 deprecated functions:
  - `get_secovi()`
  - `get_bcb_realestate()`
  - `get_abrainc_indicators()`
  - `get_abecip_indicators()`
  - `get_rppi_bis()`
  - `get_bcb_series()`
  - `get_fgv_ibre()`
  - `get_nre_ire()`

- **Removed**: Verbose `@section` blocks (Progress Reporting, Error Handling)
- **Simplified**: `@details` sections to 1-3 essential lines
- **Enhanced**: `@section Deprecation` blocks with code migration examples

#### Impact
- ~260-290 lines of documentation removed
- Documentation now focuses on **migration guidance** rather than usage examples
- All functions still exported and callable (no functionality changes)
- Help pages now emphasize using `get_dataset()` instead

#### Migration
These functions were deprecated in v0.4.0. Users should migrate to the modern API:

```r
# Old way (still works, but no longer documented with examples):
data <- get_secovi()

# New way (recommended):
data <- get_dataset("secovi")
```

Full migration examples are available in each function's `@section Deprecation` block.

#### Rationale
- **Pre-1.0.0**: Breaking changes are acceptable before stable release
- **Codebase simplification**: Reduces maintenance burden and package size
- **Focus on modern API**: Encourages users to adopt `get_dataset()` interface
- **Clear migration path**: Enhanced deprecation warnings guide users to new API

---

# realestatebr 0.5.1

## Bug Fixes

### SECOVI Default Table Fix
**Fixed SECOVI dataset to return all categories by default instead of only "condo"**

- **Problem**: `get_dataset("secovi")` was only returning the "condo" category (1,939 rows) instead of all categories (9,398 rows). This caused test failures for launch/rent/sale tables.

- **Root Cause**: When no table parameter was specified, the code defaulted to the first category alphabetically ("condo"), rather than fetching all categories.

- **Solution**:
  - Added `default_table` configuration support in `datasets.yaml`
  - Updated `validate_and_resolve_table()` to check for `default_table` setting
  - Set SECOVI's `default_table: "all"` in registry
  - Regenerated cache with all 4 categories

- **Impact**:
  - Cache size: 12KB → 55KB (includes all categories)
  - Data completeness: 1,939 → 9,398 rows
  - Categories: condo (1,939), launch (780), rent (2,779), sale (3,900)

```r
# Now returns all categories by default
get_dataset("secovi")  # → 9,398 rows, 4 categories ✅

# Specific tables still work correctly
get_dataset("secovi", "launch")  # → 780 rows
get_dataset("secovi", "rent")    # → 2,779 rows
get_dataset("secovi", "sale")    # → 3,900 rows
```

### Test Infrastructure Improvements
- Updated test suite to use `devtools::load_all()` instead of `library()` to ensure testing of development version
- Added comprehensive pre-release test suite (`tests/comprehensive_check_v0.5.qmd`)
- Added test result documentation (`tests/TEST_RESULTS_SUMMARY.md`, `tests/QUICK_SUMMARY.md`)

### Pipeline Improvements
- Updated `_targets.R` to always load development version for consistency
- Ensures targets pipeline uses latest code during cache regeneration

---

# realestatebr 0.5.0

## BREAKING CHANGES: User-Level Caching Architecture

### Major Architectural Change
**Version 0.5.0 introduces user-level caching, removing bundled datasets from the package to comply with CRAN's 5MB size limit. This is a BREAKING CHANGE that affects how datasets are accessed.**

### What Changed
- **Removed**: All cached datasets from `inst/cached_data/` (previously ~25MB)
- **Added**: User-level cache directory at `~/.local/share/realestatebr/` (Linux/Mac) or `%LOCALAPPDATA%/realestatebr/Cache/` (Windows)
- **Added**: GitHub Releases integration for pre-processed datasets
- **Changed**: `source="cache"` now refers to user cache, not package cache
- **Changed**: `source="github"` now downloads from GitHub releases, not package files

### New Cache Behavior
```r
# First use: downloads from GitHub releases to user cache
data <- get_dataset("abecip")  # Downloads once

# Subsequent uses: loads from user cache (instant, offline)
data <- get_dataset("abecip")  # Loads from ~/.local/share/realestatebr/

# Force fresh download from original source
data <- get_dataset("abecip", source = "fresh")  # Downloads and caches

# Explicit source selection
data <- get_dataset("abecip", source = "cache")   # User cache only
data <- get_dataset("abecip", source = "github")  # GitHub releases only
```

### Auto Fallback Strategy (source = "auto", default)
1. **User Cache**: Check `~/.local/share/realestatebr/` (instant, offline)
2. **GitHub Releases**: Download pre-processed data (requires `piggyback` package)
3. **Fresh Download**: Download from original source (saves to user cache)

### New Dependencies
- **Added**: `rappdirs` (Imports) - Cross-platform user cache directory support
- **Added**: `piggyback` (Suggests) - GitHub releases download support

### New Functions
- `get_user_cache_dir()`: Get path to user cache directory
- `list_cached_files()`: List all cached datasets
- `clear_user_cache()`: Remove cached datasets
- `is_cached()`: Check if dataset is in cache
- `list_github_assets()`: List available datasets on GitHub releases
- `download_from_github_release()`: Download specific dataset from releases
- `update_cache_from_github()`: Update cached datasets from GitHub
- `is_cache_up_to_date()`: Compare local vs GitHub cache timestamps

### Migration Guide

#### For Users
```r
# Install updated package
install.packages("realestatebr")  # or devtools::install_github()

# Install piggyback for GitHub downloads (recommended)
install.packages("piggyback")

# First use after update: will download datasets to user cache
data <- get_dataset("abecip")

# Check cache location
get_user_cache_dir()

# Manage cache
list_cached_files()           # See what's cached
clear_user_cache("abecip")    # Clear specific dataset
clear_user_cache()            # Clear all (with confirmation)
```

#### For Package Developers
- Cached data files now excluded from package build via `.Rbuildignore`
- Package size reduced from ~25MB to <5MB (CRAN compliant)
- `inst/cached_data/` kept for development/CI but excluded from distribution
- GitHub Actions workflow publishes cache to releases via `data-raw/publish-cache.R`

### Benefits
- ✅ **CRAN Compliant**: Package size now <5MB (was 25MB)
- ✅ **Faster Installation**: Package downloads are much smaller
- ✅ **Offline Usage**: Once cached, datasets work offline
- ✅ **User Control**: Users manage their own cache
- ✅ **Weekly Updates**: GitHub releases updated automatically by CI
- ✅ **No Breaking APIs**: `get_dataset()` interface unchanged

### Deprecations
- `import_cached()`: Still works but now loads from user cache (previously from `inst/`)
- Old `cached=TRUE` parameter in legacy functions: Still supported but uses new cache

### Files Changed
- **New**: `R/cache-user.R` - User cache management
- **New**: `R/cache-github.R` - GitHub releases integration
- **New**: `data-raw/publish-cache.R` - Upload cache to releases
- **Updated**: `R/get-dataset.R` - Refactored cache logic
- **Updated**: `R/cache.R` - Marked as deprecated (kept for compatibility)
- **Updated**: `.Rbuildignore` - Exclude `inst/cached_data/` files
- **Updated**: `DESCRIPTION` - Added `rappdirs` and `piggyback` dependencies

---

## Targets Pipeline Fixes

### Critical Pipeline Functionality
- **Fixed**: Targets pipeline now fully functional for automated data updates
- **Fixed**: FGV IBRE and NRE-IRE datasets now work correctly in targets pipeline
  - Changed from `source="fresh"` to `source="github"` for manually-updated datasets
  - These datasets have no API/download capability and require manual updates
- **Fixed**: Removed broken internal data object fallback in `get_fgv_ibre()` and `get_nre_ire()`
  - Previously tried to access non-existent `fgv_data` and `ire` objects from `R/sysdata.rda`
  - Now provides clear error messages when fresh downloads are attempted with `cached=FALSE`

### Enhanced Dataset Registry
- **Added**: `manual_update` flag to `datasets.yaml` for FGV IBRE and NRE-IRE
- **Added**: `update_notes` field documenting why fresh downloads aren't available
- **Improved**: Clear documentation in `_targets.R` explaining data source choices

### Files Changed
- `_targets.R`: Updated `fetch_dataset()` to support `source` parameter; FGV and NRE-IRE now use `source="github"`
- `R/get_fgv_ibre.R`: Removed broken internal data fallback; added clear error for fresh downloads
- `R/get_nre_ire.R`: Removed broken internal data fallback; added clear error for fresh downloads
- `inst/extdata/datasets.yaml`: Added manual update flags and notes

## Bug Fixes from Recent Commits

### Property Records Simplification (Commit 9eab0ca)
- **Refactored**: Major simplification of `get_property_records.R` (14% code reduction: 780→673 lines)
- **Removed**: Deprecated functions `get_ri_capitals()` and `get_ri_aggregates()` with warning messages
- **Removed**: Unused metadata attributes (`source`, `download_time`, `download_info`) that were never used
- **Simplified**: Documentation for internal function (removed verbose examples and sections)
- **Improved**: `scrape_registro_imoveis_links()` with better connection cleanup and reduced complexity

### BCB Dataset Critical Fixes (Commit bb580c8)

#### BCB Real Estate
- **Fixed**: CLI message serialization error in targets pipeline
- **Fixed**: Compute `nrow()` before CLI interpolation to avoid closure issues

#### BCB Series - Graceful Degradation (CRITICAL)
- **Fixed**: Replaced batch download with individual series downloads for better reliability
- **Fixed**: Now returns successful series even if some fail (e.g., 14/15 instead of 0/15)
- **Added**: Per-series retry logic with exponential backoff using `purrr::possibly()` pattern
- **Added**: Clear warnings showing which series failed
- **Restored**: Commented-out table filtering logic - now filters by `bcb_category` when `table` specified
- **Improved**: Metadata-driven approach using `bcb_metadata` dynamically (now downloads all 140 series, not just 15)

#### Get Dataset Infrastructure
- **Fixed**: BCB Real Estate table filtering by category in `get-dataset.R`
- **Fixed**: BCB Series table filtering by `bcb_category`
- **Added**: Support for `table="all"` in `validate_and_resolve_table()` function
- **Fixed**: Proper mapping of user-facing table names to internal Portuguese categories

#### Registry and Tests
- **Updated**: `bcb_series` categories in `datasets.yaml` to match metadata
- **Added**: Missing categories: production, interest-rate, exchange, government, real-estate
- **Added**: Integration tests for BCB table filtering and graceful degradation
- **Result**: All 97 integration tests now pass

### Get Dataset Critical Fixes (Commit ce4768b)

#### CLI Message Scoping
- **Fixed**: Added `.envir = parent.frame()` to `cli::cli_inform()` calls in `cli_user()` and `cli_debug()`
- **Fixed**: "cannot coerce type 'closure' to vector of type 'character'" error
- **Affected**: Previously failed for rppi_bis, property_records, and all functions using these helpers

#### FipeZap Data Quality
- **Fixed**: Added `standardize_city_names()` call after binding FipeZap data
- **Fixed**: Now correctly shows "Brazil" instead of "Índice Fipezap" for national index

#### Property Records Table Extraction
- **Fixed**: Added special handling for nested `property_records` structure in `get-dataset.R`
- **Fixed**: Now returns single tibbles instead of nested lists
- **Fixed**: All tables (capitals, cities, aggregates, transfers) now work correctly

#### Testing Infrastructure
- **Added**: Comprehensive integration test suite with 37 tests covering critical `get_dataset()` functionality
- **Added**: Tests with `source="fresh"` to catch real-world failures before production
- **Added**: GitHub Actions CI workflow for weekly integration tests
- **Added**: Manual testing script `tests/basic_checks.R` for development

### Note on Vignettes
- Vignettes temporarily set to `eval=FALSE` for faster development
- **TODO**: Re-enable vignette evaluation before CRAN release

---

# realestatebr 0.4.1

## Bug Fixes

### RPPI Individual Table Access
- **Fixed**: `get_dataset("rppi", "ivgr")` and other individual RPPI tables now work correctly
- **Fixed**: Vignette build errors caused by RPPI table routing issues
- **Improved**: Internal `get_rppi()` function now supports all individual RPPI tables (fipezap, ivgr, igmi, iqa, iqaiw, ivar, secovi_sp) in addition to stacked tables (sale, rent, all)

### CRAN Compliance
- **Fixed**: Removed all non-ASCII characters from R source files (7 files affected)
- Replaced Portuguese characters with Unicode escapes for CRAN compliance
- Files updated: `get_bcb_realestate.R`, `get_cbic.R`, `get_fgv_ibre.R`, `get_property_records.R`, `get_rppi.R`, `get_rppi_bis.R`, `get_secovi.R`

### Test Suite
- **Fixed**: Updated deprecated `category=` parameter to `table=` in `tests/sanity_check.R`

---

# realestatebr 0.4.0

## Major Breaking Changes - API Consolidation

### 🎯 Unified Data Interface

This release implements a **major breaking change** that consolidates 15+ individual `get_*()` functions into a single, unified `get_dataset()` interface. This dramatically simplifies the package API while maintaining full functionality.

**BREAKING CHANGE**: All individual `get_*()` functions have been removed:
- `get_abecip_indicators()`, `get_abrainc_indicators()`, `get_rppi()`, `get_bcb_realestate()`, etc.
- **Migration**: Use `get_dataset("dataset_name")` instead

### 🔧 RPPI Code Simplification (Internal)

**Major refactoring** of RPPI functions for better maintainability:
- **67% code reduction**: 1579 lines → 519 lines (1060 lines removed)
- **Bug fix**: FipeZap national index now correctly standardized to `name_muni == "Brazil"`
- **Shared helpers**: Created `rppi-helpers.R` with common functions to eliminate duplication
- **Removed overhead**: Eliminated unused `stack` parameter, cli_debug calls, and metadata attributes
- **Simplified documentation**: Removed verbose sections (Progress Reporting, Error Handling, Examples) from internal functions
- **All functions now `@keywords internal`**: Only `get_dataset()` is user-facing

**Benefits**:
- Easier to maintain and debug
- Faster execution (less overhead)
- Consistent error handling across all indices
- Bug fixes apply to all functions automatically

### 📊 CBIC Dataset - Partial Release (Cement Only)

**Note**: In v0.4.0, the CBIC dataset is limited to **cement tables only** (validated data). Steel and PIM tables will be added in v0.4.1.

**Available in v0.4.0**:
- ✅ `cement_monthly_consumption` - Monthly cement consumption by state
- ✅ `cement_annual_consumption` - Annual cement consumption by region
- ✅ `cement_production_exports` - Production, consumption, and export data
- ✅ `cement_monthly_production` - Monthly cement production by state
- ✅ `cement_cub_prices` - CUB cement prices by state

**Coming in v0.4.1**:
- ⏳ Steel prices and production data
- ⏳ PIM industrial production indices

```r
# Works in v0.4.0
get_dataset("cbic")  # Default: cement_monthly_consumption
get_dataset("cbic", "cement_cub_prices")

# Will error with informative message
get_dataset("cbic", "steel_prices")  # Deferred to v0.4.1
```

### New Internal Architecture

- **Internal fetch functions**: Created 12 new `fetch_*()` functions with `@keywords internal`
- **Registry-driven**: All datasets managed through centralized `inst/extdata/datasets.yaml`
- **Hierarchical RPPI**: Consolidated `rppi` and `rppi_indices` into single hierarchical structure
- **Consistent parameters**: All internal functions use standardized `table`, `cached`, `quiet`, `max_retries`

### 📋 Simplified Public API

**New unified interface:**
```r
# Get data from any dataset
data <- get_dataset("abecip")               # Default table
data <- get_dataset("abecip", table = "sbpe")  # Specific table
data <- get_dataset("rppi", table = "fipezap")  # Hierarchical access

# Discover datasets
datasets <- list_datasets()
info <- get_dataset_info("rppi")
```

**Removed functions (now internal):**
- `get_abecip_indicators()` → `get_dataset("abecip")`
- `get_abrainc_indicators()` → `get_dataset("abrainc")`
- `get_rppi()` → `get_dataset("rppi")`
- `get_bcb_realestate()` → `get_dataset("bcb_realestate")`
- `get_bcb_series()` → `get_dataset("bcb_series")`
- Plus 10 more functions

### 🔧 Enhanced Data Access

- **Smart fallback**: Auto fallback from GitHub cache → fresh download
- **Source control**: Explicit `source = "cache"/"github"/"fresh"` options
- **Better error messages**: Detailed troubleshooting information
- **Metadata preservation**: All data includes source tracking and download info

### 🧪 Comprehensive Testing

- **New test suite**: `test-internal-functions-0.4.0.R` with 100 tests
- **Registry validation**: Ensures all datasets have proper internal function mappings
- **Parameter consistency**: Validates all internal functions follow standard interface
- **Hierarchical testing**: Comprehensive RPPI access pattern validation

## Migration Guide

### For Existing Code (Breaking Changes)

```r
# OLD (0.3.x) - Will no longer work
data <- get_abecip_indicators(table = "sbpe")
data <- get_rppi(table = "fipezap")
data <- get_bcb_realestate(table = "all")

# NEW (0.4.0) - Required migration
data <- get_dataset("abecip", table = "sbpe")
data <- get_dataset("rppi", table = "fipezap")
data <- get_dataset("bcb_realestate", table = "all")
```

### Dataset Name Mapping

| Old Function | New get_dataset() Name |
|-------------|---------------------|
| `get_abecip_indicators()` | `"abecip"` |
| `get_abrainc_indicators()` | `"abrainc"` |
| `get_rppi()` | `"rppi"` |
| `get_bcb_realestate()` | `"bcb_realestate"` |
| `get_bcb_series()` | `"bcb_series"` |
| `get_rppi_bis()` | `"rppi_bis"` |
| `get_secovi()` | `"secovi"` |
| `get_fgv_indicators()` | `"fgv_indicators"` |
| `get_b3_stocks()` | `"b3_stocks"` |
| `get_nre_ire()` | `"nre_ire"` |
| `get_cbic_*()` | `"cbic"` |
| `get_itbi()` | `"itbi"` |
| `get_property_records()` | `"registro"` |

### RPPI Consolidation

```r
# OLD - Multiple functions
fipezap <- get_rppi_fipezap()
igmi <- get_rppi_igmi()
bis <- get_rppi_bis()

# NEW - Unified hierarchical access
fipezap <- get_dataset("rppi", table = "fipezap")
igmi <- get_dataset("rppi", table = "igmi")
bis <- get_dataset("rppi", table = "bis")
```

## Technical Implementation

### Internal Architecture
- **12 internal fetch functions**: `fetch_rppi()`, `fetch_abecip()`, etc.
- **Registry system**: Complete mapping in `datasets.yaml`
- **Fallback mechanism**: `get_from_internal_function()` → `get_from_legacy_function()`
- **NAMESPACE cleanup**: Only exports `get_dataset()`, `list_datasets()`, utilities

### Backward Compatibility
- **Phase 1**: Internal functions call legacy functions for gradual transition
- **Testing**: Comprehensive test coverage ensures functionality preservation
- **Error handling**: Graceful degradation with informative error messages

---

*This release represents a major architectural shift toward a unified, maintainable API. While it introduces breaking changes, the new interface is significantly simpler and more powerful.*

**Full Changelog**: https://github.com/viniciusoike/realestatebr/compare/v0.3.0...v0.4.0

---

# realestatebr 0.3.0

## Major Features and Improvements

### 🎯 Phase 2: Data Pipeline Implementation Complete

- **{targets} Pipeline Framework**: Implemented comprehensive targets workflow for automated data processing and validation
- **Automated Data Workflows**: Added daily and weekly GitHub Actions workflows using the targets pipeline
- **Data Validation Infrastructure**: Added comprehensive validation rules and reporting for all datasets
- **Pipeline Performance Monitoring**: Added automated report generation and validation status tracking

### 📊 Enhanced Data Processing

- **Targets Pipeline**: `_targets.R` workflow with automated dependency management and parallel processing
- **Validation System**: Comprehensive data validation rules with automated quality checks
- **Pipeline Helpers**: Centralized helper functions for consistent data processing across all sources
- **Report Generation**: Automated pipeline status reports and data quality summaries

### 🔧 Improved Function Reliability

- **Error Handling**: Enhanced error handling in `cache.R` with better fallback mechanisms
- **Function Fixes**: Fixed parameter bugs in `get_abrainc_indicators()` (category → table)
- **Data Access**: Improved `get_nre_ire()` to use internal package data directly
- **Internal Data**: Updated `sysdata.rda` with latest processed datasets

### 🚀 Infrastructure Improvements

- **Workflow Automation**: Replaced single update workflow with focused daily/weekly pipelines
- **Cache Management**: Improved cache validation and fallback strategies
- **Data Source Updates**: Enhanced FGV data cleaning with improved formatting
- **Dependency Updates**: Added `targets` and `tarchetypes` to package dependencies

### 📈 New Data Sources

- **B3 Stocks**: Added enhanced B3 stock data processing with standardized formatting
- **FGV Indicators**: Improved FGV consultation data processing and validation
- **Industrial Production**: Enhanced CBIC PIM data integration
- **Construction Materials**: Updated CBIC cement and steel data processing

## Technical Implementation

### Targets Pipeline Architecture
- **Automated Processing**: All datasets now processed through unified targets pipeline
- **Quality Assurance**: Built-in validation and quality checks for all data sources
- **Performance Monitoring**: Real-time pipeline status and error reporting
- **Dependency Management**: Automatic detection of data updates and re-processing

### Enhanced Error Handling
- **Graceful Degradation**: Improved fallback mechanisms for failed data retrievals
- **Better Diagnostics**: Enhanced error messages and troubleshooting information
- **Retry Logic**: Smart retry mechanisms with exponential backoff
- **Progress Reporting**: Real-time progress updates during long-running operations

### Data Quality Improvements
- **Validation Rules**: Comprehensive validation for all datasets
- **Metadata Tracking**: Enhanced metadata preservation and source tracking
- **Format Standardization**: Consistent data formatting across all sources
- **Quality Metrics**: Automated quality assessment and reporting

## Migration Notes

### For Existing Users
- All existing functions continue to work unchanged
- Enhanced reliability and performance with new pipeline backend
- Improved error messages and troubleshooting information
- Better cache management and fallback strategies

### For Developers
- New targets pipeline provides foundation for custom data workflows
- Enhanced validation framework for quality assurance
- Standardized helper functions for consistent data processing
- Comprehensive pipeline documentation and examples

---

*This release establishes the foundation for automated data processing and validation, setting the stage for Phase 3 implementation with large dataset support.*

**Full Changelog**: https://github.com/viniciusoike/realestatebr/compare/v0.2.0...v0.3.0

---

# realestatebr 0.2.0

## Major Features and Improvements

### 🎯 Phase 1 Modernization Complete

- **Modernized 13 core `get_*` functions** with consistent APIs, CLI-based error handling, and progress reporting
- **Standardized function signatures** with `table`, `cached`, `quiet`, and `max_retries` parameters
- **Robust error handling** with retry logic, exponential backoff, and informative error messages
- **Enhanced documentation** with comprehensive examples and @section blocks

### 📊 New Unified Data Architecture

- **`list_datasets()`** - Discover available datasets with filtering capabilities
- **`get_dataset()`** - Unified data access function with intelligent fallback
- **Registry system** in `inst/extdata/datasets.yaml` for centralized dataset management
- **Improved caching** with standardized cache location and validation

### 🔧 API Standardization

- **Introduced `table` parameter** replacing `category` across all functions
- **Backward compatibility maintained** with deprecation warnings for `category` parameter
- **Consistent return types** - single tibble by default, list when `table = "all"`
- **Metadata attributes** on all returned data with source tracking and download info

### 📈 New Data Sources

- **CBIC construction materials data**:
  - `get_cbic_cement()` - Cement consumption, production, and CUB prices
  - `get_cbic_steel()` - Steel prices and production data
  - `get_cbic_pim()` - Industrial production indices
- **Enhanced RPPI suite** with improved coordination and error handling
- **Updated B3 stock data** with standardized column names

### 🚀 Performance and Reliability

- **Progress reporting** with `cli` package integration for long-running operations
- **Exponential backoff** for failed web scraping and API calls
- **Parallel processing support** in web scraping functions
- **Comprehensive input validation** with helpful error messages

### 🌐 Bilingual Support

- **Translation system** for Portuguese/English column names and values
- **Standardized naming conventions** across all datasets
- **Region and state name translations** for geographic data

## Breaking Changes

### Parameter Changes
- **`category` parameter deprecated** across all functions in favor of `table`
  - Backward compatibility maintained with deprecation warnings
  - Will be removed in a future version
  - Migration: Replace `category = "value"` with `table = "value"`

### Cache Location
- **Cache moved** from `cached_data/` to `inst/cached_data/` for package compliance
- Existing cache files automatically migrated

## Modernized Functions

### Fully Modernized (13 functions)
- `get_abecip_indicators()` - ABECIP real estate financing data
- `get_abrainc_indicators()` - ABRAINC launches and sales data
- `get_b3_stocks()` - B3 stock market data with improved column naming
- `get_bcb_realestate()` - Central Bank real estate credit data
- `get_bcb_series()` - BCB macroeconomic time series
- `get_rppi_bis()` - Bank for International Settlements RPPI data
- `get_cbic_cement()` - CBIC cement industry data (NEW)
- `get_cbic_steel()` - CBIC steel industry data (NEW)
- `get_cbic_pim()` - CBIC industrial production data (NEW)
- `get_fgv_indicators()` - FGV construction confidence indicators
- `get_nre_ire()` - Real Estate Index from NRE-Poli USP
- `get_property_records()` - Property registration data with robust Excel processing
- `get_rppi()` - Comprehensive RPPI coordinator with all sources
- `get_secovi()` - SECOVI-SP real estate data with parallel processing

### Legacy Functions (Maintained)
- `get_rppi_bis()` - Main function with modernized backend and single tibble returns
- `get_itbi()` and `get_itbi_bhe()` - Planned for Phase 3 (DuckDB integration)

## Infrastructure Improvements

### New Architecture Components
- **Dataset registry system** with YAML configuration
- **Unified cache management** with validation and fallback
- **Translation framework** for multilingual support
- **Helper function ecosystem** for robust web operations

### Developer Experience
- **Comprehensive test suite** with 35+ tests covering all modernized functions
- **Consistent documentation patterns** with @section blocks and examples
- **CLI-based development workflow** with `devtools` integration
- **GitHub Actions** for automated testing and deployment

## Migration Guide

### For Existing Code
```r
# Old (deprecated but still works)
data <- get_abecip_indicators(category = "all")

# New (recommended)
data <- get_abecip_indicators(table = "all")
```

### For New Code
```r
# Discover available datasets
datasets <- list_datasets()

# Get data with unified interface
data <- get_dataset("abecip_indicators")

# Use modernized functions with progress
data <- get_abecip_indicators(table = "indicators", quiet = FALSE)
```

## Technical Details

### Dependencies
- **Added**: `cli` for modern error handling and progress reporting
- **Enhanced**: Better integration with `dplyr`, `readr`, `httr`, and `rvest`
- **Maintained**: Full backward compatibility with existing dependencies

### Performance
- **Improved web scraping** with intelligent retry logic
- **Faster cache access** with optimized file structure
- **Better memory usage** with streaming and lazy loading where appropriate

---

*This release represents the completion of Phase 1 modernization, establishing a solid foundation for Phase 2 (data pipeline automation) and Phase 3 (large dataset support with DuckDB).*

**Full Changelog**: https://github.com/viniciusoike/realestatebr/compare/v0.1.5...v0.2.0
