Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

## [Unreleased]

- Add `Region#has_code?`, `Region#has_name?`, `Region#has_handle?`, `Region#neighbor?` and `Region#valid_zone_zip?`, for matching a zone by code, by name and by zip, and `Util.utf8_handle` and `Util.ascii_casecmp_substring?`, which they use. `valid_zone_zip?` checks a province's own `zip_prefixes`. `valid_zip?` is unchanged: it infers one province from the zip, so it can differ for a shared prefix or a zip shorter than a prefix. [#682](https://github.com/Shopify/worldwide/pull/682)
- Add the `postal:codes:import`, `postal:codes:check` and `postal:codes:zone_check` rake tasks. They build and check the `zip_prefixes` and `zips_crossing_provinces` data in `data/regions/<CC>.yml`. [#680](https://github.com/Shopify/worldwide/pull/680)

---
Expand Down
198 changes: 195 additions & 3 deletions lib/worldwide/region.rb
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ class Region
).to_set.freeze
private_constant :ISO_3166_ASSIGNED_COUNTRY_CODES

CODE_SEPARATOR = "-"
private_constant :CODE_SEPARATOR

BLANK_CODE = /\A[[:space:]]*\z/
private_constant :BLANK_CODE

# Accpetable format types of the postal code.
FORMAT_TYPES = {
ALPHANUMERIC: "ALPHANUMERIC",
Expand Down Expand Up @@ -164,10 +170,10 @@ class Region

# Other names that may be used to refer to this region.
# E.g., "Czech Republic" is also known as "Czechia".
attr_accessor :name_alternates
attr_reader :name_alternates

# iso_code values of regions (subdivisions) within the same country that border this region.
# E.g., for CA-ON, the neighbouring zones are CA-MB, CA-NU and CA-QC.
# legacy_code values of regions (subdivisions) within the same country that border this region.
# E.g., for CA-ON, the neighbouring zones are "QC" and "MB".
attr_accessor :neighbours

alias_method :neighbors, :neighbours
Expand Down Expand Up @@ -336,6 +342,12 @@ def initialize(
@zones_by_code = {}
end

def name_alternates=(names)
@name_alternates = names
@handle = nil
@handle_alternates = nil
end

def inspect
"#<#{self.class.name}:#{object_id} #{inspected_fields}>"
end
Expand Down Expand Up @@ -557,6 +569,115 @@ def valid_zip?(zip, partial_match: false)
valid_normalized_zip?(normalized, partial_match: partial_match)
end

# Is the given postal code value valid for this region, judging a province by its own zip prefixes?
#
# For a region that is not a province, this gives the same result as {#valid_zip?}.
#
# For a province, {#valid_zip?} infers one province from the zip, and accepts the zip only when that is
# this province or when `zips_crossing_provinces` lists it. This method instead accepts the zip when
# the country accepts it and one of this province's own `zip_prefixes` matches it. The two differ when:
#
# - the province has no `zip_prefixes`, but other provinces of its country do: this method accepts
# any zip that the country accepts, and {#valid_zip?} accepts only a zip that infers this province;
# - the prefixes of two provinces both match the zip, for example a partial zip such as "CH6" or a
# shared prefix such as "K1A": this method accepts the zip for each of them, and {#valid_zip?}
# accepts it for the first one that it finds, and for the others only through `zips_crossing_provinces`.
#
# @example
# Worldwide.region(code: "CA-ON").valid_zone_zip?("K2A") == false
# Worldwide.region(code: "CA-ON").valid_zone_zip?("K2A", partial_match: true) == true
#
# @param zip [String]
# @param partial_match [Boolean] also accept the first part of a zip, as `valid_zip?` does
# @return [Boolean]
def valid_zone_zip?(zip, partial_match: false)
return valid_zip?(zip, partial_match: partial_match) unless province?

country = associated_country
return false if country.nil? || country.iso_code.nil?

country_code = country.iso_code
normalized = Zip.normalize(country_code: country_code, zip: zip)
adjusted = Zip.strip_optional_country_prefix(country_code: country_code, zip: normalized)

return false unless country.valid_zip?(adjusted, partial_match: partial_match)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits: adjusted is already normalized here, but country.valid_zip? normalizes it again. Checking adjusted directly would skip the repeat.
The return false if zip.nil? below also looks unreachable: a blank zip would always return here.

return true if Util.blank?(zip) && !country.zip_required?
return true if Util.blank?(zip_prefixes)
return false if zip.nil?
return true if country.send(:cross_border_zip_includes_province?, zip: adjusted, province_code: legacy_code)

zip_prefixes.any? { |prefix| adjusted.start_with?(prefix) || prefix.start_with?(adjusted) }
end

# Does the given code identify this region?
#
# The code may be the region's `legacy_code`, one of its `code_alternates`, or its `iso_code`.
# For a region within a country, the code may also carry the country's code in place of
# the hyphen-less part: "CA-NL" matches the region with legacy code "NL" in Canada.
# And, for a region whose legacy code has a hyphen, the part after the hyphen alone is enough:
# "13" matches the region with legacy code "JP-13".
#
# The comparison ignores the case of ASCII letters.
# Callers may look a region up by code very often, so this allocates no strings.
#
# @example
# Worldwide.region(code: "CA-QC").has_code?("PQ") == true
# Worldwide.region(code: "CA-QC").has_code?("ca-qc") == true
# Worldwide.region(code: "JP-13").has_code?("13") == true
#
# @param code [String, Symbol]
# @return [Boolean]
def has_code?(code)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This byte matching path (match_unprefixed_code?, suffix_matches?, match_suffix_only_code? and Util.ascii_casecmp_substring?) is a lot of code for a few short ASCII codes. Could we build each region's accepted codes once and compare them with casecmp? In a quick check across all zones it gave the same results.

return false if code.nil?

code = code.to_s unless code.is_a?(String)
return false if code.empty? || BLANK_CODE.match?(code)

return true if legacy_code&.casecmp(code)&.zero?
return true if code_alternates&.any? { |code_alternate| code_alternate.casecmp(code)&.zero? }
return true if iso_code&.casecmp(code)&.zero?

match_unprefixed_code?(code) || match_suffix_only_code?(code)
end

# Does the given name identify this region?
#
# The name is compared with the region's `legacy_name` and its `name_alternates`, after
# {Util.utf8_handle} simplifies each of them, so that "Fayyum" matches "Fayyūm".
# It is not compared with the localized {#full_name}.
#
# @param name [String]
# @return [Boolean]
def has_name?(name)
has_handle?(name.nil? ? nil : Util.utf8_handle(name))
end

# Does the given handle identify this region?
#
# A handle is the result of {Util.utf8_handle}. It is compared with the handle of the region's
# `legacy_name` and with the handles of its `name_alternates`.
#
# @param handle [String]
# @return [Boolean]
def has_handle?(handle)
return false if handle.nil?

handle == name_handle || name_alternate_handles.include?(handle)
end

# Is the region with the given legacy code one of the regions that border this region?
#
# @example
# Worldwide.region(code: "CA-ON").neighbor?("QC") == true
#
# @param code [String] a legacy code, as found in {#neighbours}
# @return [Boolean]
def neighbor?(code)
return false if neighbours.nil?

neighbours.include?(code)
end

# are zones optional for this region?
def province_optional?
province_optional || !@zones&.any?(&:province?)
Expand All @@ -571,6 +692,77 @@ def has_zip_prefixes?

private

def name_handle
return if legacy_name.nil?

@handle ||= Util.utf8_handle(legacy_name)
end

def name_alternate_handles
@handle_alternates ||= (name_alternates || []).map { |name| Util.utf8_handle(name) }.freeze
end

# Matches a code that has this region's country code, a hyphen, and then this region's legacy code
# or one of its code alternates. E.g., "CA-NL", to match just the "NL" part.
#
# This avoids String#split, so that it allocates no strings.
def match_unprefixed_code?(code)
return false unless province?

hyphen_index = code.byteindex(CODE_SEPARATOR)
return false unless hyphen_index && hyphen_index != 0 && hyphen_index + 1 < code.bytesize

country_code = parent_country&.iso_code
return false if country_code.nil? || country_code.empty?

return false unless Util.ascii_casecmp_substring?(
country_code,
code,
start1: 0,
length1: country_code.bytesize,
start2: 0,
length2: hyphen_index,
)

suffix_start = hyphen_index + 1
suffix_length = code.bytesize - suffix_start

return true if suffix_matches?(legacy_code, code, suffix_start, suffix_length)

!!code_alternates&.any? { |code_alternate| suffix_matches?(code_alternate, code, suffix_start, suffix_length) }
end

def suffix_matches?(candidate, code, suffix_start, suffix_length)
return false if candidate.nil? || candidate.empty?

Util.ascii_casecmp_substring?(
candidate,
code,
start1: 0,
length1: candidate.bytesize,
start2: suffix_start,
length2: suffix_length,
)
end

# Matches a code that is only the part of this region's legacy code after the hyphen,
# e.g., "13" for "JP-13". Code alternates do not take part, to avoid matching by accident.
def match_suffix_only_code?(code)
return false if legacy_code.nil?

hyphen_index = legacy_code.byteindex(CODE_SEPARATOR)
return false if hyphen_index.nil? || hyphen_index == legacy_code.bytesize - 1

Util.ascii_casecmp_substring?(
legacy_code,
code,
start1: hyphen_index + 1,
length1: legacy_code.bytesize - hyphen_index - 1,
start2: 0,
length2: code.bytesize,
)
end

def add_zone_to_hash(zone)
@zones_by_code[subdivision_code(zone.iso_code)] = zone if Util.present?(subdivision_code(zone.iso_code))
@zones_by_code[zone.alpha_three] = zone if Util.present?(zone.alpha_three)
Expand Down
101 changes: 101 additions & 0 deletions lib/worldwide/util.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,40 @@

module Worldwide
class Util
# Characters in U+00C0..U+017F that have no decomposition into a base letter and a mark,
# mapped to an ASCII approximation.
HANDLE_APPROXIMATIONS = {
"ß" => "ss",
"æ" => "ae",
"ð" => "d",
"ø" => "o",
"þ" => "th",
"đ" => "d",
"ħ" => "h",
"ı" => "i",
"ij" => "ij",
"ĸ" => "k",
"ŀ" => "l",
"ł" => "l",
"ʼn" => "'n",
"ŋ" => "ng",
"œ" => "oe",
"ŧ" => "t",
}.freeze
private_constant :HANDLE_APPROXIMATIONS

HANDLE_LATIN_RANGE = /[\u00C0-\u017F]/
private_constant :HANDLE_LATIN_RANGE

HANDLE_QUOTES_AND_BRACKETS = "'\"()[]"
private_constant :HANDLE_QUOTES_AND_BRACKETS

HANDLE_SEPARATOR = /(?:(?!\p{Word}|\u200D|\p{Extended_Pictographic}|\p{Emoji_Modifier}).)+/m
private_constant :HANDLE_SEPARATOR

HANDLE_EDGE_HYPHENS = /\A-+|-+\z/
private_constant :HANDLE_EDGE_HYPHENS

class << self
def blank?(value)
value.nil? ||
Expand All @@ -12,6 +46,73 @@ def blank?(value)
def present?(value)
!blank?(value)
end

# Reduces a name to a simplified, lowercase form that is suitable for comparing names.
# Latin letters with diacritics become their base letters, quotes and brackets are dropped, and each
# run of other punctuation or whitespace becomes a single hyphen.
# Letters of other scripts are kept.
#
# @example
# Worldwide::Util.utf8_handle("Al Fayyūm") == "al-fayyum"
# Worldwide::Util.utf8_handle("Ærøskøbing 日本") == "aeroskobing-日本"
#
# @param input [String]
# @return [String] a new, frozen string
def utf8_handle(input)
string = input.to_s
string = string.dup.force_encoding(Encoding::UTF_8) unless string.encoding == Encoding::UTF_8
string = ActiveSupport::Multibyte::Unicode.tidy_bytes(string) unless string.valid_encoding?

result = string.downcase.unicode_normalize(:nfc)
result = result.gsub(HANDLE_LATIN_RANGE) do |char|
HANDLE_APPROXIMATIONS[char] || char.unicode_normalize(:nfd).delete("\u0300-\u036F")
end
result.delete!(HANDLE_QUOTES_AND_BRACKETS)
result.gsub!(HANDLE_SEPARATOR, "-")
result.gsub!(HANDLE_EDGE_HYPHENS, "")
result.freeze
end

# Compares a part of one string with a part of another string, ignoring the case of ASCII letters.
# This allocates no objects, because it compares bytes with `getbyte`.
# A comparison that involves a byte outside of ASCII is always false.
#
# @param str1 [String]
# @param str2 [String]
# @param start1 [Integer] the byte offset of the part of `str1`
# @param length1 [Integer] the byte length of the part of `str1`; it must be greater than 0
# @param start2 [Integer] the byte offset of the part of `str2`
# @param length2 [Integer] the byte length of the part of `str2`; it must be greater than 0
# @return [Boolean]
# @raise [ArgumentError] when a string is nil, a length is not positive, or a start is negative
def ascii_casecmp_substring?(str1, str2, start1:, length1:, start2:, length2:)
raise ArgumentError, "str1 and str2 must not be nil" if str1.nil? || str2.nil?
raise ArgumentError, "Length must be greater than 0" if length1 <= 0 || length2 <= 0
raise ArgumentError, "Start indices must not be negative" if start1 < 0 || start2 < 0

return false if length1 != length2

length = length1
return false if start1 + length > str1.bytesize || start2 + length > str2.bytesize

# A while loop avoids allocating a Range
i = 0
while i < length
b1 = str1.getbyte(start1 + i)
b2 = str2.getbyte(start2 + i)

return false if b1 >= 128 || b2 >= 128

b1 += 32 if b1.between?(65, 90)
b2 += 32 if b2.between?(65, 90)

return false if b1 != b2

i += 1
end

true
end
end
end
end
Loading
Loading