dawarich/app/models/import.rb

88 lines
2.3 KiB
Ruby
Raw Normal View History

# frozen_string_literal: true
2024-03-15 18:27:31 -04:00
class Import < ApplicationRecord
belongs_to :user
2024-03-15 18:27:31 -04:00
has_many :points, dependent: :destroy
2025-03-23 13:37:10 -04:00
has_one_attached :file
2025-06-30 14:29:47 -04:00
# Flag to skip background processing during user data import
attr_accessor :skip_background_processing
after_commit -> { Import::ProcessJob.perform_later(id) unless skip_background_processing }, on: :create
after_commit :remove_attached_file, on: :destroy
before_commit :recalculate_stats, on: :destroy, if: -> { points.exists? }
validates :name, presence: true, uniqueness: { scope: :user_id }
validate :file_size_within_limit, if: -> { user.trial? }
validate :import_count_within_limit, if: -> { user.trial? }
0.37.0 (#2067) * fix: move foreman to global gems to fix startup crash (#1971) * Update exporting code to stream points data to file in batches to red… (#1980) * Update exporting code to stream points data to file in batches to reduce memory usage * Update changelog * Update changelog * Feature/maplibre frontend (#1953) * Add a plan to use MapLibre GL JS for the frontend map rendering, replacing Leaflet * Implement phase 1 * Phases 1-3 + part of 4 * Fix e2e tests * Phase 6 * Implement fog of war * Phase 7 * Next step: fix specs, phase 7 done * Use our own map tiles * Extract v2 map logic to separate manager classes * Update settings panel on v2 map * Update v2 e2e tests structure * Reimplement location search in maps v2 * Update speed routes * Implement visits and places creation in v2 * Fix last failing test * Implement visits merging * Fix a routes e2e test and simplify the routes layer styling. * Extract js to modules from maps_v2_controller.js * Implement area creation * Fix spec problem * Fix some e2e tests * Implement live mode in v2 map * Update icons and panel * Extract some styles * Remove unused file * Start adding dark theme to popups on MapLibre maps * Make popups respect dark theme * Move v2 maps to maplibre namespace * Update v2 references to maplibre * Put place, area and visit info into side panel * Update API to use safe settings config method * Fix specs * Fix method name to config in SafeSettings and update usages accordingly * Add missing public files * Add handling for real time points * Fix remembering enabled/disabled layers of the v2 map * Fix lots of e2e tests * Add settings to select map version * Use maps/v2 as main path for MapLibre maps * Update routing * Update live mode * Update maplibre controller * Update changelog * Remove some console.log statements * Pull only necessary data for map v2 points * Feature/raw data archive (#2009) * 0.36.2 (#2007) * fix: move foreman to global gems to fix startup crash (#1971) * Update exporting code to stream points data to file in batches to red… (#1980) * Update exporting code to stream points data to file in batches to reduce memory usage * Update changelog * Update changelog * Feature/maplibre frontend (#1953) * Add a plan to use MapLibre GL JS for the frontend map rendering, replacing Leaflet * Implement phase 1 * Phases 1-3 + part of 4 * Fix e2e tests * Phase 6 * Implement fog of war * Phase 7 * Next step: fix specs, phase 7 done * Use our own map tiles * Extract v2 map logic to separate manager classes * Update settings panel on v2 map * Update v2 e2e tests structure * Reimplement location search in maps v2 * Update speed routes * Implement visits and places creation in v2 * Fix last failing test * Implement visits merging * Fix a routes e2e test and simplify the routes layer styling. * Extract js to modules from maps_v2_controller.js * Implement area creation * Fix spec problem * Fix some e2e tests * Implement live mode in v2 map * Update icons and panel * Extract some styles * Remove unused file * Start adding dark theme to popups on MapLibre maps * Make popups respect dark theme * Move v2 maps to maplibre namespace * Update v2 references to maplibre * Put place, area and visit info into side panel * Update API to use safe settings config method * Fix specs * Fix method name to config in SafeSettings and update usages accordingly * Add missing public files * Add handling for real time points * Fix remembering enabled/disabled layers of the v2 map * Fix lots of e2e tests * Add settings to select map version * Use maps/v2 as main path for MapLibre maps * Update routing * Update live mode * Update maplibre controller * Update changelog * Remove some console.log statements --------- Co-authored-by: Robin Tuszik <mail@robin.gg> * Remove esbuild scripts from package.json * Remove sideEffects field from package.json * Raw data archivation * Add tests * Fix tests * Fix tests * Update ExceptionReporter * Add schedule to run raw data archival job monthly * Change file structure for raw data archival feature * Update changelog and version for raw data archival feature --------- Co-authored-by: Robin Tuszik <mail@robin.gg> * Set raw_data to an empty hash instead of nil when archiving * Fix storage configuration and file extraction * Consider MIN_MINUTES_SPENT_IN_CITY during stats calculation (#2018) * Consider MIN_MINUTES_SPENT_IN_CITY during stats calculation * Remove raw data from visited cities api endpoint * Use user timezone to show dates on maps (#2020) * Fix/pre epoch time (#2019) * Use user timezone to show dates on maps * Limit timestamps to valid range to prevent database errors when users enter pre-epoch dates. * Limit timestamps to valid range to prevent database errors when users enter pre-epoch dates. * Fix tests failing due to new index on stats table * Fix failing specs * Update redis client configuration to support unix socket connection * Update changelog * Fix kml kmz import issues (#2023) * Fix kml kmz import issues * Refactor KML importer to improve readability and maintainability * Implement moving points in map v2 and fix route rendering logic to ma… (#2027) * Implement moving points in map v2 and fix route rendering logic to match map v1. * Fix route spec * fix(maplibre): update date format to ISO 8601 (#2029) * Add verification step to raw data archival process (#2028) * Add verification step to raw data archival process * Add actual verification of raw data archives after creation, and only clear raw_data for verified archives. * Fix failing specs * Eliminate zip-bomb risk * Fix potential memory leak in js * Return .keep files * Use Toast instead of alert for notifications * Add help section to navbar dropdown * Update changelog * Remove raw_data_archival_job * Ensure file is being closed properly after reading in Archivable concern * Add composite index to stats table if not exists * Update changelog * Update entrypoint to always sync static assets (not only new ones) * Add family layer to MapLibre maps (#2055) * Add family layer to MapLibre maps * Update migration * Don't show family toggle if feature is disabled * Update changelog * Return changelog * Update changelog * Update tailwind file * Bump sentry-rails from 6.0.0 to 6.1.0 (#1945) Bumps [sentry-rails](https://github.com/getsentry/sentry-ruby) from 6.0.0 to 6.1.0. - [Release notes](https://github.com/getsentry/sentry-ruby/releases) - [Changelog](https://github.com/getsentry/sentry-ruby/blob/master/CHANGELOG.md) - [Commits](https://github.com/getsentry/sentry-ruby/compare/6.0.0...6.1.0) --- updated-dependencies: - dependency-name: sentry-rails dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump turbo-rails from 2.0.17 to 2.0.20 (#1944) Bumps [turbo-rails](https://github.com/hotwired/turbo-rails) from 2.0.17 to 2.0.20. - [Release notes](https://github.com/hotwired/turbo-rails/releases) - [Commits](https://github.com/hotwired/turbo-rails/compare/v2.0.17...v2.0.20) --- updated-dependencies: - dependency-name: turbo-rails dependency-version: 2.0.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Evgenii Burmakin <Freika@users.noreply.github.com> * Bump webmock from 3.25.1 to 3.26.1 (#1943) Bumps [webmock](https://github.com/bblimke/webmock) from 3.25.1 to 3.26.1. - [Release notes](https://github.com/bblimke/webmock/releases) - [Changelog](https://github.com/bblimke/webmock/blob/master/CHANGELOG.md) - [Commits](https://github.com/bblimke/webmock/compare/v3.25.1...v3.26.1) --- updated-dependencies: - dependency-name: webmock dependency-version: 3.26.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Evgenii Burmakin <Freika@users.noreply.github.com> * Bump brakeman from 7.1.0 to 7.1.1 (#1942) Bumps [brakeman](https://github.com/presidentbeef/brakeman) from 7.1.0 to 7.1.1. - [Release notes](https://github.com/presidentbeef/brakeman/releases) - [Changelog](https://github.com/presidentbeef/brakeman/blob/main/CHANGES.md) - [Commits](https://github.com/presidentbeef/brakeman/compare/v7.1.0...v7.1.1) --- updated-dependencies: - dependency-name: brakeman dependency-version: 7.1.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump redis from 5.4.0 to 5.4.1 (#1941) Bumps [redis](https://github.com/redis/redis-rb) from 5.4.0 to 5.4.1. - [Changelog](https://github.com/redis/redis-rb/blob/master/CHANGELOG.md) - [Commits](https://github.com/redis/redis-rb/compare/v5.4.0...v5.4.1) --- updated-dependencies: - dependency-name: redis dependency-version: 5.4.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Put import deletion into background job (#2045) * Put import deletion into background job * Update changelog * fix null type error and update heatmap styling (#2037) * fix: use constant weight for maplibre heatmap layer * fix null type, update heatmap styling * improve heatmap styling * fix typo * Fix stats calculation to recursively reduce H3 resolution when too ma… (#2065) * Fix stats calculation to recursively reduce H3 resolution when too many hexagons are generated * Update CHANGELOG.md * Validate trip start and end dates (#2066) * Validate trip start and end dates * Update changelog * Update migration to clean up duplicate stats before adding unique index * Fix fog of war radius setting being ignored and applying settings causing errors (#2068) * Update changelog * Add Rack::Deflater middleware to config/application.rb to enable gzip compression for responses. * Add composite index to points on user_id and timestamp * Deduplicte points based on timestamp brought to unix time * Fix/stats cache invalidation (#2072) * Fix family layer toggle in Map v2 settings for non-selfhosted env * Invalidate cache * Remove comments * Remove comment * Add new indicies to improve performance and remove unused ones to opt… (#2078) * Add new indicies to improve performance and remove unused ones to optimize database. * Remove comments * Update map search suggestions panel styling * Add yearly digest (#2073) * Add yearly digest * Rename YearlyDigests to Users::Digests * Minor changes * Update yearly digest layout and styles * Add flags and chart to email * Update colors * Fix layout of stats in yearly digest view * Remove cron job for yearly digest scheduling * Update CHANGELOG.md * Update digest email setting handling * Allow sharing digest for 1 week or 1 month * Change Digests Distance to Bigint * Fix settings page * Update changelog * Add RailsPulse (#2079) * Add RailsPulse * Add RailsPulse monitoring tool with basic HTTP authentication * Bring points_count to integer * Update migration and version * Update rubocop issues --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Robin Tuszik <mail@robin.gg> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-30 11:33:56 -05:00
enum :status, { created: 0, processing: 1, completed: 2, failed: 3, deleting: 4 }
2025-06-28 06:22:56 -04:00
2024-09-05 15:01:59 -04:00
enum :source, {
google_semantic_history: 0, owntracks: 1, google_records: 2,
2025-06-28 06:22:56 -04:00
google_phone_takeout: 3, gpx: 4, immich_api: 5, geojson: 6, photoprism_api: 7,
0.36.0 (#1952) * Implement OmniAuth GitHub authentication * Fix omniauth GitHub scope to include user email access * Remove margin-bottom * Implement Google OAuth2 authentication * Implement OIDC authentication for Dawarich using omniauth_openid_connect gem. * Add patreon account linking and patron checking service * Update docker-compose.yml to use boolean values instead of strings * Add support for KML files * Add tests * Update changelog * Remove patreon OAuth integration * Move omniauthable to a concern * Update an icon in integrations * Update changelog * Update app version * Fix family location sharing toggle * Move family location sharing to its own controller * Update changelog * Implement basic tagging functionality for places, allowing users to categorize and label places with custom tags. * Add places management API and tags feature * Add some changes related to places management feature * Fix some tests * Fix sometests * Add places layer * Update places layer to use Leaflet.Control.Layers.Tree for hierarchical layer control * Rework tag form * Add hashtag * Add privacy zones to tags * Add notes to places and manage place tags * Update changelog * Update e2e tests * Extract tag serializer to its own file * Fix some tests * Fix tags request specs * Fix some tests * Fix rest of the tests * Revert some changes * Add missing specs * Revert changes in place export/import code * Fix some specs * Fix PlaceFinder to only consider global places when finding existing places * Fix few more specs * Fix visits creator spec * Fix last tests * Update place creating modal * Add home location based on "Home" tagged place * Save enabled tag layers * Some fixes * Fix bug where enabling place tag layers would trigger saving enabled layers, overwriting with incomplete data * Update migration to use disable_ddl_transaction! and add up/down methods * Fix tag layers restoration and filtering logic * Update OIDC auto-registration and email/password registration settings * Fix potential xss
2025-11-24 13:45:09 -05:00
user_data_archive: 8, kml: 9
2025-08-23 10:07:15 -04:00
}, allow_nil: true
def process!
2025-06-28 06:22:56 -04:00
if user_data_archive?
process_user_data_archive!
else
Imports::Create.new(user, self).call
end
end
def process_user_data_archive!
Users::ImportDataJob.perform_later(id)
end
2024-12-06 10:52:36 -05:00
def reverse_geocoded_points_count
points.reverse_geocoded.count
end
2024-12-06 10:52:36 -05:00
def years_and_months_tracked
2024-12-10 12:49:37 -05:00
points.order(:timestamp).pluck(:timestamp).map do |timestamp|
time = Time.zone.at(timestamp)
[time.year, time.month]
2024-12-06 10:52:36 -05:00
end.uniq
end
def migrate_to_new_storage
return if file.attached?
raw_file = File.new(raw_data)
file.attach(io: raw_file, filename: name, content_type: 'application/json')
end
2025-06-30 16:29:28 -04:00
private
def remove_attached_file
file.purge_later
end
2025-08-19 12:55:22 -04:00
def file_size_within_limit
return unless file.attached?
return unless file.blob.byte_size > 11.megabytes
errors.add(:file, 'is too large. Trial users can only upload files up to 10MB.')
end
def import_count_within_limit
return unless new_record?
existing_imports_count = user.imports.count
return unless existing_imports_count >= 5
2025-09-23 15:16:25 -04:00
errors.add(:base, 'Trial users can only create up to 5 imports. Please subscribe to import more files.')
end
def recalculate_stats
years_and_months_tracked.each do |year, month|
Stats::CalculatingJob.perform_later(user.id, year, month)
2025-08-19 12:55:22 -04:00
end
end
2024-03-15 18:27:31 -04:00
end