diff --git a/.superdesign/design_iterations/trip_page_1.html b/.superdesign/design_iterations/trip_page_1.html deleted file mode 100644 index fb29fe20..00000000 --- a/.superdesign/design_iterations/trip_page_1.html +++ /dev/null @@ -1,283 +0,0 @@ - - - - - - European Grand Tour - Trip Details - - - -
- -
-

- European Grand Tour -

-

- A 21-day journey through the heart of Europe, discovering historic cities, stunning landscapes, and rich cultural heritage. -

-
- - -
- -
-
-
- -
-
-

Interactive Map

-

Route visualization would appear here

-
- - -
-
- Start: Amsterdam -
-
-
-
- End: Rome -
-
-
-
-
- - -
- -
-

Trip Statistics

- -
-
-
3,247 km
-
Total Distance
-
- -
-
21 days
-
Duration
-
- -
-
7 countries
-
Countries Visited
-
-
-
- - -
-

Countries Visited

-
-
- Netherlands - 3 days -
-
- Germany - 4 days -
-
- Austria - 2 days -
-
- Switzerland - 3 days -
-
- France - 4 days -
-
- Monaco - 1 day -
-
- Italy - 4 days -
-
-
-
-
- - -
-
-

Trip Photos

-
147 photos
-
- - -
- -
-
-
-
-
-
-
-

Amsterdam Canal

-
-
- -
-
-
-
-
-
-
-

Berlin Wall

-
-
- -
-
-
-
-
-
-
-

Alpine Vista

-
-
- -
-
-
-
-
-
-
-

Swiss Mountains

-
-
- -
-
-
-
-
-
-
-

Eiffel Tower

-
-
- -
-
-
-
-
-
-
-

Monaco Harbor

-
-
- -
-
-
-
-
-
-
-

Colosseum

-
-
- -
-
-
-
-
-
-
-

Roman Forum

-
-
- - -
- -
-
-
- - -
-

Trip Timeline

- -
-
-
-
-
-
Day 1-3: Amsterdam, Netherlands
-
Explored canals, visited museums, experienced local culture
-
-
- -
-
-
-
Day 4-7: Berlin & Munich, Germany
-
Historical sites, traditional cuisine, alpine preparation
-
-
- -
-
-
-
Day 8-9: Salzburg, Austria
-
Mozart's birthplace, stunning architecture
-
-
- -
-
-
-
Day 10-12: Zurich & Alps, Switzerland
-
Mountain adventures, pristine lakes, scenic drives
-
-
- -
-
-
-
Day 13-16: Paris & Lyon, France
-
Art, cuisine, romance, and French countryside
-
-
- -
-
-
-
Day 17: Monaco
-
Luxury, casinos, and Mediterranean coastline
-
-
- -
-
-
-
Day 18-21: Rome, Italy
-
Ancient history, incredible food, perfect ending
-
-
-
-
-
-
- - \ No newline at end of file diff --git a/.superdesign/design_iterations/trip_page_2.html b/.superdesign/design_iterations/trip_page_2.html deleted file mode 100644 index bd2133b6..00000000 --- a/.superdesign/design_iterations/trip_page_2.html +++ /dev/null @@ -1,238 +0,0 @@ - - - - - - Asian Adventure - Trip Details - - - - -
- - -
-

- Asian Adventure -

-

- A journey through Southeast Asia's cultural treasures -

-
- - -
- - -
-
-
-
-

Interactive Map

-

Route visualization

-
-
-
- - -
- - -
-

Trip Statistics

- -
-
-
2,847 km
-
Total Distance
-
- -
-
18 days
-
Duration
-
- -
-
5 countries
-
Countries Visited
-
-
-
- - -
-

Countries

-
-
- Thailand - 6 days -
-
- Vietnam - 4 days -
-
- Cambodia - 3 days -
-
- Laos - 3 days -
-
- Myanmar - 2 days -
-
-
- - -
-

Highlights

-
-
-
- 12 temples visited -
-
-
- 4 cooking classes -
-
-
- 8 markets explored -
-
-
- 3 boat rides -
-
-
-
-
- - -
-
-

Trip Photos

- 247 photos -
- - -
- -
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- - -
- -
-
- - -
-

Trip Timeline

- -
-
-
- Day 1-6 -
-
-

Bangkok & Northern Thailand

-

- Explored the bustling streets of Bangkok, visited ancient temples, and trekked through the mountains of Chiang Mai. -

-
-
- -
-
- Day 7-10 -
-
-

Ho Chi Minh City & Hanoi

-

- Discovered Vietnamese culture, cuisine, and history across the country's two major cities. -

-
-
- -
-
- Day 11-13 -
-
-

Siem Reap, Cambodia

-

- Marveled at the ancient temples of Angkor Wat and experienced traditional Khmer culture. -

-
-
- -
-
- Day 14-16 -
-
-

Luang Prabang, Laos

-

- Experienced the peaceful atmosphere of this UNESCO World Heritage city along the Mekong River. -

-
-
- -
-
- Day 17-18 -
-
-

Yangon, Myanmar

-

- Concluded the journey with visits to golden pagodas and local markets in Myanmar's largest city. -

-
-
-
-
-
- - \ No newline at end of file diff --git a/.superdesign/design_iterations/trip_page_3.html b/.superdesign/design_iterations/trip_page_3.html deleted file mode 100644 index 8e635fcf..00000000 --- a/.superdesign/design_iterations/trip_page_3.html +++ /dev/null @@ -1,316 +0,0 @@ - - - - - - Coast to Coast Adventure - Trip Details - - - - - -
-
- - -
-

- Coast to Coast Adventure -

-

- New York City to San Francisco • October 2024 -

-
- - -
- - -
-
-
-

Route Overview

-

Interactive journey across America

-
-
-
-
- - - - -
-

Interactive Map

-
-
-
-
- - -
-
-

Trip Statistics

- -
- -
-
-
-

Total Distance

-

2,908 mi

-
-
- - - -
-
-
- - -
-
-
-

Duration

-

14 days

-
-
- - - -
-
-
- - -
-
-
-

States Visited

-

12

-
-
- - - -
-
-
-
- - -
-

States Crossed

-
-
-
- New York -
-
-
- Pennsylvania -
-
-
- Ohio -
-
-
- Indiana -
-
-
- Illinois -
-
-
- Iowa -
-
-
- Nebraska -
-
-
- Colorado -
-
-
- Utah -
-
-
- Nevada -
-
-
- California -
-
-
-
-
- - -
-
-

Trip Highlights

- -
- -
-
-
-
- - - -
-

Golden Gate Bridge

-
-
-
- - -
-
-
- - - -
-

Chicago Skyline

-
-
- -
-
-
- - - -
-

Rocky Mountains

-
-
- -
-
-
- - - -
-

Monument Valley

-
-
- -
-
-
- - - -
-

Route 66

-
-
-
- - -
- -
-
-
-
- - -
- -
-

Key Stops

-
-
- Times Square, NYC - Day 1 -
-
- Millennium Park, Chicago - Day 4 -
-
- Rocky Mountain National Park - Day 8 -
-
- Arches National Park - Day 10 -
-
- Golden Gate Bridge, SF - Day 14 -
-
-
- - -
-

Weather Summary

-
-
- Average Temperature - 68°F -
-
- Sunny Days - 11 of 14 -
-
- Rain Days - 2 of 14 -
-
- Best Weather - Utah, Nevada -
-
-
- - -
-

Trip Notes

-
-

Perfect timing for fall foliage in the Midwest. Colorado mountains were breathtaking with early snow caps.

-

Route 66 sections in Illinois and Missouri provided authentic American road trip experience.

-

Utah's landscape diversity exceeded expectations - from desert to mountain passes.

-
-
-
-
-
- - \ No newline at end of file diff --git a/.superdesign/design_iterations/trip_page_3_1.html b/.superdesign/design_iterations/trip_page_3_1.html deleted file mode 100644 index b50ad622..00000000 --- a/.superdesign/design_iterations/trip_page_3_1.html +++ /dev/null @@ -1,189 +0,0 @@ - - - - - - Coast to Coast Adventure - Trip Details - - - - - -
-
- - -
-
-
-

Coast to Coast Adventure

-

NYC → SF • Oct 2024

-
-
-
2,908 mi
-
14 days
-
-
-
- - -
- - -
-
-
-
-
- - - - -
-

Route Map

-
-
-
-
- - -
- -
-

Trip Stats

-
-
- Distance - 2,908 mi -
-
- Duration - 14 days -
-
- States - 12 -
-
- Photos - 247 -
-
-
- - -
-

Route

-
- NY → PA → OH → IN → IL → IA → NE → CO → UT → NV → CA -
-
- - -
-

Highlights

-
-
- - - -
-
- - - -
-
- - - -
-
- - - -
-
-
-
-
- - -
- -
-

Key Stops

-
-
- Times Square - Day 1 -
-
- Chicago - Day 4 -
-
- Rocky Mountains - Day 8 -
-
- Arches NP - Day 10 -
-
- Golden Gate - Day 14 -
-
-
- - -
-

Weather

-
-
- Avg Temp - 68°F -
-
- Sunny Days - 11/14 -
-
- Rain Days - 2/14 -
-
- Best - Utah, Nevada -
-
-
- - -
-

Notes

-
-

Fall foliage in Midwest was perfect timing.

-

Route 66 sections provided authentic experience.

-

Utah landscape diversity exceeded expectations.

-
-
-
-
-
- - \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index a24e2f08..bf30ec65 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - [ ] In the User Settings, you can now import your user data from a zip file. It will import all the data from the zip file, listed above. It will also start stats recalculation. - [ ] User can select to override settings or not. + - [ ] Check distance units if they are correct + - [ ] Why import creates more points than the original? - Export file size is now displayed in the exports and imports lists. diff --git a/app/models/point.rb b/app/models/point.rb index e4d7b0eb..44dbc68d 100644 --- a/app/models/point.rb +++ b/app/models/point.rb @@ -29,7 +29,7 @@ class Point < ApplicationRecord scope :visited, -> { where.not(visit_id: nil) } scope :not_visited, -> { where(visit_id: nil) } - after_create :async_reverse_geocode, if: -> { DawarichSettings.store_geodata? } + after_create :async_reverse_geocode, if: -> { DawarichSettings.store_geodata? && !reverse_geocoded? } after_create :set_country after_create_commit :broadcast_coordinates diff --git a/app/services/users/import_data/points.rb b/app/services/users/import_data/points.rb index a33427d1..1f472169 100644 --- a/app/services/users/import_data/points.rb +++ b/app/services/users/import_data/points.rb @@ -1,6 +1,8 @@ # frozen_string_literal: true class Users::ImportData::Points + BATCH_SIZE = 1000 + def initialize(user, points_data) @user = user @points_data = points_data @@ -11,83 +13,81 @@ class Users::ImportData::Points Rails.logger.info "Importing #{points_data.size} points for user: #{user.email}" - points_created = 0 - skipped_invalid = 0 + # Pre-load reference data for efficient bulk processing + preload_reference_data + + # Filter valid points and prepare for bulk import + valid_points = filter_and_prepare_points + + if valid_points.empty? + Rails.logger.info "No valid points to import" + return 0 + end + + # Remove duplicates based on unique constraint + deduplicated_points = deduplicate_points(valid_points) + + Rails.logger.info "Prepared #{deduplicated_points.size} unique valid points (#{points_data.size - deduplicated_points.size} duplicates/invalid skipped)" + + # Bulk import in batches + total_created = bulk_import_points(deduplicated_points) + + Rails.logger.info "Points import completed. Created: #{total_created}" + total_created + end + + private + + attr_reader :user, :points_data, :imports_lookup, :countries_lookup, :visits_lookup + + def preload_reference_data + # Pre-load imports for this user + @imports_lookup = user.imports.index_by { |import| + [import.name, import.source, import.created_at.to_s] + } + + # Pre-load all countries for efficient lookup + @countries_lookup = {} + Country.all.each do |country| + # Index by all possible lookup keys + @countries_lookup[[country.name, country.iso_a2, country.iso_a3]] = country + @countries_lookup[country.name] = country + end + + # Pre-load visits for this user + @visits_lookup = user.visits.index_by { |visit| + [visit.name, visit.started_at.to_s, visit.ended_at.to_s] + } + end + + def filter_and_prepare_points + valid_points = [] + skipped_count = 0 points_data.each do |point_data| next unless point_data.is_a?(Hash) # Skip points with invalid or missing required data unless valid_point_data?(point_data) - skipped_invalid += 1 + skipped_count += 1 next end - # Check if point already exists (match by coordinates, timestamp, and user) - if point_exists?(point_data) + # Prepare point attributes for bulk insert + prepared_attributes = prepare_point_attributes(point_data) + unless prepared_attributes + skipped_count += 1 next end - # Create new point - point_record = create_point_record(point_data) - points_created += 1 if point_record - - if points_created % 1000 == 0 - Rails.logger.debug "Imported #{points_created} points..." - end + valid_points << prepared_attributes end - if skipped_invalid > 0 - Rails.logger.warn "Skipped #{skipped_invalid} points with invalid or missing required data" + if skipped_count > 0 + Rails.logger.warn "Skipped #{skipped_count} points with invalid or missing required data" end - Rails.logger.info "Points import completed. Created: #{points_created}" - points_created - end - - private - - attr_reader :user, :points_data - - def point_exists?(point_data) - return false unless point_data['lonlat'].present? && point_data['timestamp'].present? - - Point.exists?( - lonlat: point_data['lonlat'], - timestamp: point_data['timestamp'], - user_id: user.id - ) - rescue StandardError => e - Rails.logger.debug "Error checking if point exists: #{e.message}" - false - end - - def create_point_record(point_data) - point_attributes = prepare_point_attributes(point_data) - - begin - # Create point and skip the automatic country assignment callback since we're handling it manually - point = Point.create!(point_attributes) - - # If we have a country assigned via country_info, update the point to set it - if point_attributes[:country].present? - point.update_column(:country_id, point_attributes[:country].id) - point.reload - end - - point - rescue ActiveRecord::RecordInvalid => e - Rails.logger.error "Failed to create point: #{e.message}" - Rails.logger.error "Point data: #{point_data.inspect}" - Rails.logger.error "Prepared attributes: #{point_attributes.inspect}" - nil - rescue StandardError => e - Rails.logger.error "Unexpected error creating point: #{e.message}" - Rails.logger.error "Point data: #{point_data.inspect}" - Rails.logger.error "Prepared attributes: #{point_attributes.inspect}" - Rails.logger.error "Backtrace: #{e.backtrace.first(5).join('\n')}" - nil - end + valid_points end def prepare_point_attributes(point_data) @@ -99,68 +99,124 @@ class Users::ImportData::Points 'country_info', 'visit_reference', 'country' # Exclude the string country field - handled via country_info relationship - ).merge(user: user) + ) # Handle lonlat reconstruction if missing (for backward compatibility) ensure_lonlat_field(attributes, point_data) - # Find and assign related records - assign_import_reference(attributes, point_data['import_reference']) - assign_country_reference(attributes, point_data['country_info']) - assign_visit_reference(attributes, point_data['visit_reference']) + # Remove longitude/latitude after lonlat reconstruction to ensure consistent keys + attributes.delete('longitude') + attributes.delete('latitude') - attributes + # Add required attributes for bulk insert + attributes['user_id'] = user.id + attributes['created_at'] = Time.current + attributes['updated_at'] = Time.current + + # Resolve foreign key relationships + resolve_import_reference(attributes, point_data['import_reference']) + resolve_country_reference(attributes, point_data['country_info']) + resolve_visit_reference(attributes, point_data['visit_reference']) + + # Convert string keys to symbols for consistency with Point model + attributes.symbolize_keys + rescue StandardError => e + Rails.logger.error "Failed to prepare point attributes: #{e.message}" + Rails.logger.error "Point data: #{point_data.inspect}" + nil end - def assign_import_reference(attributes, import_reference) + def resolve_import_reference(attributes, import_reference) return unless import_reference.is_a?(Hash) - import = user.imports.find_by( - name: import_reference['name'], - source: import_reference['source'], - created_at: import_reference['created_at'] - ) + import_key = [ + import_reference['name'], + import_reference['source'], + import_reference['created_at'] + ] - attributes[:import] = import if import + import = imports_lookup[import_key] + attributes['import_id'] = import.id if import end - def assign_country_reference(attributes, country_info) + def resolve_country_reference(attributes, country_info) return unless country_info.is_a?(Hash) # Try to find country by all attributes first - country = Country.find_by( - name: country_info['name'], - iso_a2: country_info['iso_a2'], - iso_a3: country_info['iso_a3'] - ) + country_key = [country_info['name'], country_info['iso_a2'], country_info['iso_a3']] + country = countries_lookup[country_key] # If not found by all attributes, try to find by name only if country.nil? && country_info['name'].present? - country = Country.find_by(name: country_info['name']) + country = countries_lookup[country_info['name']] end - # If still not found, create a new country record with minimal data + # If still not found, create a new country record if country.nil? && country_info['name'].present? - country = Country.find_or_create_by(name: country_info['name']) do |new_country| - new_country.iso_a2 = country_info['iso_a2'] || country_info['name'][0..1].upcase - new_country.iso_a3 = country_info['iso_a3'] || country_info['name'][0..2].upcase - new_country.geom = "MULTIPOLYGON (((0 0, 1 0, 1 1, 0 1, 0 0)))" # Default geometry + country = create_missing_country(country_info) + # Add to lookup cache for subsequent points + @countries_lookup[country_info['name']] = country + @countries_lookup[[country.name, country.iso_a2, country.iso_a3]] = country + end + + attributes['country_id'] = country.id if country + end + + def create_missing_country(country_info) + Country.find_or_create_by(name: country_info['name']) do |new_country| + new_country.iso_a2 = country_info['iso_a2'] || country_info['name'][0..1].upcase + new_country.iso_a3 = country_info['iso_a3'] || country_info['name'][0..2].upcase + new_country.geom = "MULTIPOLYGON (((0 0, 1 0, 1 1, 0 1, 0 0)))" # Default geometry + end + rescue StandardError => e + Rails.logger.error "Failed to create missing country: #{e.message}" + nil + end + + def resolve_visit_reference(attributes, visit_reference) + return unless visit_reference.is_a?(Hash) + + visit_key = [ + visit_reference['name'], + visit_reference['started_at'], + visit_reference['ended_at'] + ] + + visit = visits_lookup[visit_key] + attributes['visit_id'] = visit.id if visit + end + + def deduplicate_points(points) + points.uniq { |point| [point[:lonlat], point[:timestamp], point[:user_id]] } + end + + def bulk_import_points(points) + total_created = 0 + + points.each_slice(BATCH_SIZE) do |batch| + begin + # Use upsert_all to efficiently bulk insert/update points + result = Point.upsert_all( + batch, + unique_by: %i[lonlat timestamp user_id], + returning: %w[id], + on_duplicate: :skip + ) + + batch_created = result.count + total_created += batch_created + + Rails.logger.debug "Processed batch of #{batch.size} points, created #{batch_created}, total created: #{total_created}" + + rescue StandardError => e + Rails.logger.error "Failed to process point batch: #{e.message}" + Rails.logger.error "Batch size: #{batch.size}" + Rails.logger.error "Backtrace: #{e.backtrace.first(3).join('\n')}" + # Continue with next batch instead of failing completely end end - attributes[:country] = country if country - end - - def assign_visit_reference(attributes, visit_reference) - return unless visit_reference.is_a?(Hash) - - visit = user.visits.find_by( - name: visit_reference['name'], - started_at: visit_reference['started_at'], - ended_at: visit_reference['ended_at'] - ) - - attributes[:visit] = visit if visit + total_created end def valid_point_data?(point_data)