bellingcat/sugartrail
1
0
Fork 0
mirror of https://github.com/bellingcat/sugartrail.git synced 2026-06-11 21:18:30 +03:00
Code Issues Packages Projects Releases Wiki Activity

fixed address history missing entries

Browse Source
This commit is contained in:
seangreaves
2023-01-01 18:09:35 +00:00
parent 2335be495a
commit 5bf88c8694
3 changed files with 10 additions and 451 deletions
Download Patch File Download Diff File Expand all files Collapse all files

278
dashboard/.ipynb_checkpoints/quickstart_voila-checkpoint.ipynb
View File

@@ -1,278 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b6926e35",
"metadata": {},
"source": [
"# sugartrail "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f17ebdd2",
"metadata": {},
"outputs": [],
"source": [
"from sugartrail import mapview, api, base\n",
"import ipywidgets as widgets\n",
"from IPython.display import display\n",
"import requests"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cbc5e202",
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"network = base.Network()"
]
},
{
"cell_type": "markdown",
"id": "1704e377",
"metadata": {},
"source": [
"1. Insert your [Companies House API](https://developer.company-information.service.gov.uk/how-to-create-an-application) key:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0632780b",
"metadata": {},
"outputs": [],
"source": [
"API_input = widgets.Text(\n",
" value='',\n",
" placeholder='Insert API Key',\n",
" disabled=False\n",
")\n",
"\n",
"company_text = widgets.Text(\n",
" value='',\n",
" placeholder='Insert Company ID',\n",
" disabled=True\n",
")\n",
"\n",
"auth_status = widgets.HTML(\n",
" value=\"\",\n",
")\n",
"\n",
"auth_button = widgets.Button(description='Authenticate',button_style='success')\n",
"auth_button.on_click(lambda bt: auth())\n",
"\n",
"init_button = widgets.Button(description='Initialise',button_style='success', disabled=True)\n",
"init_button.on_click(lambda bt: init_network()) \n",
"\n",
"def auth():\n",
" auth_button.disabled=True\n",
" API_input.disabled=True\n",
" api.basic_auth.username = API_input.value\n",
" if api.test():\n",
" auth_status.value = u'\\u2705: Login successful'\n",
" company_text.disabled = False\n",
" init_button.disabled = False\n",
" else:\n",
" auth_button.disabled=False\n",
" API_input.disabled=False\n",
" auth_status.value = u'\\u274c: Invalid API key'\n",
"\n",
"display(API_input, auth_button, auth_status)"
]
},
{
"cell_type": "markdown",
"id": "2bd8c5be",
"metadata": {},
"source": [
"2. Insert the unique company registration number (CRN) for a company you would like to investigate:"
]
},
{
"cell_type": "markdown",
"id": "d5f9b6ad",
"metadata": {},
"source": [
"<!-- Insert a valid [Companies House Public Data API key](https://developer.company-information.service.gov.uk/get-started/) as `username` string value below. If you don't want to use the API and would prefer loading a pre-built network, uncomment and run the cell below and then run the final cell to build and load the map. -->"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "128106c5",
"metadata": {},
"outputs": [],
"source": [
"init_status = widgets.HTML(\n",
" value=\"\",\n",
") \n",
"\n",
"depth_selector = widgets.BoundedIntText(\n",
" value=1,\n",
" min=1,\n",
" max=5,\n",
" step=1,\n",
" disabled=True\n",
")\n",
"\n",
"generate_network_button = widgets.Button(description='Build Network',button_style='success', disabled=True)\n",
"generate_network_button.on_click(lambda bt: generate_network()) \n",
"\n",
"def init_network():\n",
" init_button.disabled=True\n",
" company_text.disabled=True\n",
" api.basic_auth.username = API_input.value\n",
" response = api.get_company(str(company_text.value))\n",
" if response:\n",
" network.company_id = str(company_text.value)\n",
" init_status.value = u'\\u2705: Initialisation successful for ' + str(response['company_name']) \n",
" depth_selector.disabled = False\n",
" generate_network_button.disabled = False\n",
" else:\n",
" init_button.disabled=False\n",
" company_text.disabled=False\n",
" auth_button.disabled=False\n",
" API_input.disabled=False\n",
" init_status.value = u'\\u274c: Initialisation Failed. No records for company: ' + str(company_text.value) + ' found.'\n",
"\n",
"display(company_text, init_button, init_status)"
]
},
{
"cell_type": "markdown",
"id": "addafb36",
"metadata": {},
"source": [
"3. Select the depth of the network you would like to build:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ea0e8392",
"metadata": {},
"outputs": [],
"source": [
"build_status = widgets.HTML(\n",
" value=\"\",\n",
")\n",
"\n",
"build_map_button=widgets.Button(description='Build Map',button_style='success', disabled=True)\n",
"build_map_button.on_click(lambda bt: generate_map()) \n",
"\n",
"def generate_network():\n",
" with output_box:\n",
" depth_selector.disabled = True\n",
" generate_network_button.disabled = True\n",
" network.perform_hop(depth_selector.value + 1)\n",
" network.run_map_preprocessing()\n",
" build_map_button.disabled = False\n",
" \n",
" \n",
"output_box = widgets.Output()\n",
"display(depth_selector, generate_network_button, build_status, output_box)"
]
},
{
"cell_type": "markdown",
"id": "03ffce05",
"metadata": {},
"source": [
"4. Visualise network on a map:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6449cd96",
"metadata": {},
"outputs": [],
"source": [
"map_container = widgets.HTML(\n",
" value=\"\",\n",
")\n",
"\n",
"map_data,path_table = mapview.build_map(network, clear_widget=False) \n",
"hbox_map = widgets.HBox([path_table])\n",
"vbox_map = widgets.VBox([map_data, hbox_map])\n",
"\n",
"accordion_map = widgets.Accordion(children=[vbox_map])\n",
"accordion_map.set_title(0, 'Map')\n",
"\n",
"tabs = ['Companies', 'Addresses', 'Officers', 'Company Details']\n",
"children = [widgets.Output() for tab in tabs]\n",
"tab = widgets.Tab()\n",
"tab.children = children\n",
"for i,title in enumerate(tabs):\n",
" tab.set_title(i, title)\n",
"\n",
"accordion_data = widgets.Accordion(children=[tab])\n",
"accordion_data.set_title(0, 'Data')\n",
"\n",
"def generate_map():\n",
" map_data,path_table = mapview.build_map(network, clear_widget=False) \n",
" hbox_map = widgets.HBox([path_table])\n",
" vbox_map.children = [map_data, hbox_map]\n",
" accordion_map.selected_index=0\n",
" accordion_data.selected_index=0\n",
" build_map_button.disabled = True\n",
" with tab.children[0]:\n",
" display(network.company_ids)\n",
" with tab.children[1]:\n",
" display(network.addresses)\n",
" with tab.children[2]:\n",
" display(network.officer_ids)\n",
" with tab.children[3]:\n",
" display(network.companies) \n",
"\n",
"display(build_map_button, map_container)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "684a116e",
"metadata": {},
"outputs": [],
"source": [
"accordion_map"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1e328d41",
"metadata": {},
"outputs": [],
"source": [
"accordion_data"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

172
notebooks/.ipynb_checkpoints/quickstart-checkpoint.ipynb
View File

@@ -1,172 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b6926e35",
"metadata": {},
"source": [
"*Quickstart hands-on exercise. For in-depth intro checkout Tutorial 1:*"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f17ebdd2",
"metadata": {},
"outputs": [],
"source": [
"from sugartrail import mapview, api, base\n",
"from ipywidgets import VBox, HBox"
]
},
{
"cell_type": "markdown",
"id": "d5f9b6ad",
"metadata": {},
"source": [
"Insert a valid [Companies House Public Data API key](https://developer.company-information.service.gov.uk/get-started/) as `username` string value below. If you don't want to use the API and would prefer loading a pre-built network, uncomment and run the cell below and then run the final cell to build and load the map. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4a9639e6",
"metadata": {},
"outputs": [],
"source": [
"# # network build from Domain Foundation, company_id = \"11951034\"\n",
"# import pickle\n",
"\n",
"# with open('../assets/networks/domain_corp_network.pickle', 'rb') as handle:\n",
"# network = pickle.load(handle)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "89b0082a",
"metadata": {},
"outputs": [],
"source": [
"api.basic_auth.username = \"\""
]
},
{
"cell_type": "markdown",
"id": "63220f29",
"metadata": {},
"source": [
"Enter the company number (as string) for a company you would like to explore. Example value is provided: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8aca6a54",
"metadata": {},
"outputs": [],
"source": [
"company_id = \"11951034\"\n",
"network = base.Network(company_id=company_id)"
]
},
{
"cell_type": "markdown",
"id": "7de31e72",
"metadata": {
"tags": [
"5"
]
},
"source": [
"Perform `n` number of hops (3 or less at first is advised to keep the network manageable in size):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d80be86d",
"metadata": {
"tags": [
"6"
]
},
"outputs": [],
"source": [
"n = 3\n",
"network = base.Network(company_id=company_id)\n",
"network.perform_hop(n)"
]
},
{
"cell_type": "markdown",
"id": "4481c80d",
"metadata": {},
"source": [
"Now lets visualise the connections:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "022f026e",
"metadata": {},
"outputs": [],
"source": [
"network.run_map_preprocessing()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "01dca0cf",
"metadata": {
"scrolled": false,
"tags": [
"7"
]
},
"outputs": [],
"source": [
"map_data,path_table = mapview.build_map(network) \n",
"hbox = HBox([path_table])\n",
"vbox = VBox([map_data, hbox])\n",
"vbox"
]
},
{
"cell_type": "markdown",
"id": "457bf4d0",
"metadata": {},
"source": [
"Each marker represents a company in the network. Green markers represent active companies based at the address, red markers represent active companies no longer based at the address and black markers represent dissolved companies once based at the address. \n",
"\n",
"Select a marker to display additional information: \n",
"- pop-up with the selected company's name and address\n",
"- table containing the most efficient paths from the origin to the selected company\n",
"- antpaths for each company in the network. Red antpath represents the path through all the historic addresses for the selected company. Black antpath represents the path from the network origin through all the addresses in the path to the selected company as displayed in the table. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

11
sugartrail/processing.py
View File

@@ -111,7 +111,16 @@ def build_address_history(company_id):
addresses.append(entry)
return addresses
else:
return []
address_history = []
entry = {}
for k, key in enumerate(["date_of_creation","date_of_cessation","registered_office_address"]):
if key in company_info:
entry[address_keys[k]] = company_info[key]
else:
entry[address_keys[k]] = None
entry["company_number"] = str(company_id)
entry['address'] = normalise_address(entry['address'])
return [entry]
else:
address_history = []
entry = {}