mirror of
https://github.com/Freika/dawarich.git
synced 2026-01-09 08:47:11 -05:00
* fix: move foreman to global gems to fix startup crash (#1971) * Update exporting code to stream points data to file in batches to red… (#1980) * Update exporting code to stream points data to file in batches to reduce memory usage * Update changelog * Update changelog * Feature/maplibre frontend (#1953) * Add a plan to use MapLibre GL JS for the frontend map rendering, replacing Leaflet * Implement phase 1 * Phases 1-3 + part of 4 * Fix e2e tests * Phase 6 * Implement fog of war * Phase 7 * Next step: fix specs, phase 7 done * Use our own map tiles * Extract v2 map logic to separate manager classes * Update settings panel on v2 map * Update v2 e2e tests structure * Reimplement location search in maps v2 * Update speed routes * Implement visits and places creation in v2 * Fix last failing test * Implement visits merging * Fix a routes e2e test and simplify the routes layer styling. * Extract js to modules from maps_v2_controller.js * Implement area creation * Fix spec problem * Fix some e2e tests * Implement live mode in v2 map * Update icons and panel * Extract some styles * Remove unused file * Start adding dark theme to popups on MapLibre maps * Make popups respect dark theme * Move v2 maps to maplibre namespace * Update v2 references to maplibre * Put place, area and visit info into side panel * Update API to use safe settings config method * Fix specs * Fix method name to config in SafeSettings and update usages accordingly * Add missing public files * Add handling for real time points * Fix remembering enabled/disabled layers of the v2 map * Fix lots of e2e tests * Add settings to select map version * Use maps/v2 as main path for MapLibre maps * Update routing * Update live mode * Update maplibre controller * Update changelog * Remove some console.log statements * Pull only necessary data for map v2 points * Feature/raw data archive (#2009) * 0.36.2 (#2007) * fix: move foreman to global gems to fix startup crash (#1971) * Update exporting code to stream points data to file in batches to red… (#1980) * Update exporting code to stream points data to file in batches to reduce memory usage * Update changelog * Update changelog * Feature/maplibre frontend (#1953) * Add a plan to use MapLibre GL JS for the frontend map rendering, replacing Leaflet * Implement phase 1 * Phases 1-3 + part of 4 * Fix e2e tests * Phase 6 * Implement fog of war * Phase 7 * Next step: fix specs, phase 7 done * Use our own map tiles * Extract v2 map logic to separate manager classes * Update settings panel on v2 map * Update v2 e2e tests structure * Reimplement location search in maps v2 * Update speed routes * Implement visits and places creation in v2 * Fix last failing test * Implement visits merging * Fix a routes e2e test and simplify the routes layer styling. * Extract js to modules from maps_v2_controller.js * Implement area creation * Fix spec problem * Fix some e2e tests * Implement live mode in v2 map * Update icons and panel * Extract some styles * Remove unused file * Start adding dark theme to popups on MapLibre maps * Make popups respect dark theme * Move v2 maps to maplibre namespace * Update v2 references to maplibre * Put place, area and visit info into side panel * Update API to use safe settings config method * Fix specs * Fix method name to config in SafeSettings and update usages accordingly * Add missing public files * Add handling for real time points * Fix remembering enabled/disabled layers of the v2 map * Fix lots of e2e tests * Add settings to select map version * Use maps/v2 as main path for MapLibre maps * Update routing * Update live mode * Update maplibre controller * Update changelog * Remove some console.log statements --------- Co-authored-by: Robin Tuszik <mail@robin.gg> * Remove esbuild scripts from package.json * Remove sideEffects field from package.json * Raw data archivation * Add tests * Fix tests * Fix tests * Update ExceptionReporter * Add schedule to run raw data archival job monthly * Change file structure for raw data archival feature * Update changelog and version for raw data archival feature --------- Co-authored-by: Robin Tuszik <mail@robin.gg> * Set raw_data to an empty hash instead of nil when archiving * Fix storage configuration and file extraction * Consider MIN_MINUTES_SPENT_IN_CITY during stats calculation (#2018) * Consider MIN_MINUTES_SPENT_IN_CITY during stats calculation * Remove raw data from visited cities api endpoint * Use user timezone to show dates on maps (#2020) * Fix/pre epoch time (#2019) * Use user timezone to show dates on maps * Limit timestamps to valid range to prevent database errors when users enter pre-epoch dates. * Limit timestamps to valid range to prevent database errors when users enter pre-epoch dates. * Fix tests failing due to new index on stats table * Fix failing specs * Update redis client configuration to support unix socket connection * Update changelog * Fix kml kmz import issues (#2023) * Fix kml kmz import issues * Refactor KML importer to improve readability and maintainability * Implement moving points in map v2 and fix route rendering logic to ma… (#2027) * Implement moving points in map v2 and fix route rendering logic to match map v1. * Fix route spec * fix(maplibre): update date format to ISO 8601 (#2029) * Add verification step to raw data archival process (#2028) * Add verification step to raw data archival process * Add actual verification of raw data archives after creation, and only clear raw_data for verified archives. * Fix failing specs * Eliminate zip-bomb risk * Fix potential memory leak in js * Return .keep files * Use Toast instead of alert for notifications * Add help section to navbar dropdown * Update changelog * Remove raw_data_archival_job * Ensure file is being closed properly after reading in Archivable concern --------- Co-authored-by: Robin Tuszik <mail@robin.gg>
96 lines
2.9 KiB
Ruby
96 lines
2.9 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
module Points
|
|
module RawData
|
|
class Clearer
|
|
BATCH_SIZE = 10_000
|
|
|
|
def initialize
|
|
@stats = { cleared: 0, skipped: 0 }
|
|
end
|
|
|
|
def call
|
|
Rails.logger.info('Starting raw_data clearing for verified archives...')
|
|
|
|
verified_archives.find_each do |archive|
|
|
clear_archive_points(archive)
|
|
end
|
|
|
|
Rails.logger.info("Clearing complete: #{@stats}")
|
|
@stats
|
|
end
|
|
|
|
def clear_specific_archive(archive_id)
|
|
archive = Points::RawDataArchive.find(archive_id)
|
|
|
|
unless archive.verified_at.present?
|
|
Rails.logger.warn("Archive #{archive_id} not verified, skipping clear")
|
|
return { cleared: 0, skipped: 0 }
|
|
end
|
|
|
|
clear_archive_points(archive)
|
|
end
|
|
|
|
def clear_month(user_id, year, month)
|
|
archives = Points::RawDataArchive.for_month(user_id, year, month)
|
|
.where.not(verified_at: nil)
|
|
|
|
Rails.logger.info("Clearing #{archives.count} verified archives for #{year}-#{format('%02d', month)}...")
|
|
|
|
archives.each { |archive| clear_archive_points(archive) }
|
|
end
|
|
|
|
private
|
|
|
|
def verified_archives
|
|
# Only archives that are verified but have points with non-empty raw_data
|
|
Points::RawDataArchive
|
|
.where.not(verified_at: nil)
|
|
.where(id: points_needing_clearing.select(:raw_data_archive_id).distinct)
|
|
end
|
|
|
|
def points_needing_clearing
|
|
Point.where(raw_data_archived: true)
|
|
.where.not(raw_data: {})
|
|
.where.not(raw_data_archive_id: nil)
|
|
end
|
|
|
|
def clear_archive_points(archive)
|
|
Rails.logger.info(
|
|
"Clearing points for archive #{archive.id} " \
|
|
"(#{archive.month_display}, chunk #{archive.chunk_number})..."
|
|
)
|
|
|
|
point_ids = Point.where(raw_data_archive_id: archive.id)
|
|
.where(raw_data_archived: true)
|
|
.where.not(raw_data: {})
|
|
.pluck(:id)
|
|
|
|
if point_ids.empty?
|
|
Rails.logger.info("No points to clear for archive #{archive.id}")
|
|
return
|
|
end
|
|
|
|
cleared_count = clear_points_in_batches(point_ids)
|
|
@stats[:cleared] += cleared_count
|
|
Rails.logger.info("✓ Cleared #{cleared_count} points for archive #{archive.id}")
|
|
rescue StandardError => e
|
|
ExceptionReporter.call(e, "Failed to clear points for archive #{archive.id}")
|
|
Rails.logger.error("✗ Failed to clear archive #{archive.id}: #{e.message}")
|
|
end
|
|
|
|
def clear_points_in_batches(point_ids)
|
|
total_cleared = 0
|
|
|
|
point_ids.each_slice(BATCH_SIZE) do |batch|
|
|
Point.transaction do
|
|
Point.where(id: batch).update_all(raw_data: {})
|
|
total_cleared += batch.size
|
|
end
|
|
end
|
|
|
|
total_cleared
|
|
end
|
|
end
|
|
end
|
|
end
|