Removing Duplicated Lines by VIM

Recently I have interest in GPS location coordinates. In my TapToCount-3W iOS app, there is a feature to record location information for each tap.

I have been puzzled by a scenarior for long time. For example, if I make several taps at same location, the GPS locations spead within a range, several tens of meters to meters apart, 30 meters or 8 meters as examples.

Further invetigation, I found that the decimal points of GPS coordinates are related to location accuracy. The more decical points are in coordindates, the more accuracy.

VIM is a powerful tool. The following is a real case of my invetigation and analysis of GPS coordindates by using VIM.

Export GPS Information

First, I have to get GPS location coordates from my app. I use the Export feature to share the result of cvs file to my mac.

Using Numbers to open the csv file. The tap information includes my sleep times between Dec 2018 to Mar 2022, 7484 lines of taps I made. From GPS coordinates, I see the decimal points up to 14 digits.

For example, here is a coordinate of latitue, longitue, and altitue value:

53.55404449631790,	-113.2990255394140,	692.3739891052250

Find Duplicate GPS Information

7484 lines are quite a lot of information. I need to find out if there is any duplicated information.

Use the following command to find duplicate coordinate information:

/\(T-\d\)\(.*\)\n\(.*\2\n)\+

Match pattern explaination: first group beginning with "T-" and a digit, then any charts (coordinates) as second group, new line, next or more lines of any charts followed by the matched second group (coordinates).

The result is as followings:

Remove Duplicate Coordinate Lines

With about search result, I can then proceed removing duplicate coordinate lines.

Use the following command to remove duplicate coordinate information:

:%s/\(T-\d\)\(.*\)\(\n.*\2\)\+/\1\2/

Replacement command explaination: replace the matched pattern (duplicate GPS coordinate lines with group 1 and group 2 defined in the matched pattern.

Analysis Results

With VIM search and replacement as a convenience and powerful tool, I can do my research and analysis on GPS coordinates collected in my TapToCount-3W app.

Using Numbers, I can get coordinate values with different decimal digits, and find out what duplicate coordinates.

7,485 Taps with GPS coordindates

Decimal
Places

Degrees

Duplicates

Unique

0

0

5246

2239

1

0.1

5245

2240

2

0.01

5245

2240

3

0.001

5222

2263

4

0.000 1

4711

2774

5

0.000 01

4638

2847

14

0.000 000 000
000 00

577

6908

Based on above analysis, I think that the best decimal points should 5, 4, or 3 places, with accuracy corespoinding to +/-0.555m, +/-5.55m, or +/-55.5m. This will reduce duplicate coordinates up to 68% to 70%.

By using VIM, I did further analysis. I export a group of taps from Jan 1, 2022 to now (Mar 19, 2022), which I know for certain that they(454 taps) were tapped at the same location. The following are 2 results: one with altitute into consideration and another with no altitute.

454 Taps with GPS coordindates and altitude

Decimal
Places

Degrees

Duplicates

Unique

0

0

444

10

1

0.1

444

10

2

0.01

444

10

3

0.001

442

12

4

0.000 1

255

199

5

0.000 01

434

20

14

0.000 000 000
000 00

444

10

454 Taps with GPS coordindates and NO altitude

Decimal
Places

Degrees

Duplicates

Unique

0

0

453

1

1

0.1

453

1

2

0.01

453

1

3

0.001

452

2

4

0.000 1

444

10

5

0.000 01

33

421

14

0.000 000 000
000 00

8

446

With above research results, they helps me better to understand GPS coordindate information, and to take better and effective strategies for updating my app freatures and interfaces for my app users.

References

• wifiGPS.com: Accuracy versus decimal places
• StackExchange - GIS: Measuring accuracy of latitude and longitude?
• StackOverflow: Removing duplicate rows in vi?
• StackExchange - Vi and VIM: Delete all consecutive duplicates
• TapToCount-3W web page
• App Store: TapToCount-3W