Big Data Techniques Afford Valuable Insights into Government Data
Wednesday, 7th May 2014
By Andrew Lucas
The methodologies and analytical techniques of big data are being applied to data collected by government, either directly by the state or by commercial organisations. There are issues of privacy and fears of "big brother" relating to the way government information is used. There are also questions about the quality of big data and whether it can replace more traditional data gathering by the state. Author Andrew Lucas also highlights how information professionals can make the most of this opportunity.
The age of big data has unlocked a seemingly vast potential to make use of datasets to gain new insights and opportunities.
Driven initially by the web giants such as Google and Amazon, big data techniques have also been applied to customer data by the bricks and mortar companies, including Tesco in the UK and Walmart in the US.
Analysing State Data Leads to Valuable Insights
Big data techniques and analysis are also being applied to data collected by government in the public sphere to create valuable new insights in the areas such as public records, land & real estate data and weather data.
In the commercial world use of big data has been driven by the "digital exhaust" of commercial transactions and the ever growing "Internet of Things", but there is also a wealth of information collected by government for the purposes of public administration including:
- Company filings
- Crime fighting
- Land and property transactions
- Population/census records
As governments cut back on their spending on some of the traditional data gathering methods, such as the American Community Survey and UK census, they are turning to big data to fill the information gaps.
Commercial organisations which make use of government data in their services, for example Dun & Bradstreet and The Weather Company are also making use of big data methodologies to combine datasets and to provide greater insights into the content.
Reliability and Ethics
There are some ethical issues relating to the use of big data by government.
Whistleblower Edward Snowden revealed that the National Security Agency (NSA) in the US and the Government Communication Headquarters (GCHQ) in the UK make use of big data to monitor web traffic. Citizens may also object to the commercial use of data gathered by the state.
Questions have also been raised about the reliability of some of the claims for big data in the public sphere. Google Flu Trends, which tracks the spread of flu across the world, was held up as the exemplar of how big data can provide quicker and cheaper outcomes, but the accuracy of the findings are now being re-examined.
Big Data and Information Professionals
So why is this important to information professionals? Big data presents an opportunity for information professionals to get involved from several angles:
- Understanding where big data is in use and how it can enhance research
- Which public sources and which vendors use big data - and how
- Acting as company advisers on how to collate and analyse the enterprise's own big data
- Providing guidance on big data and intellectual property
- Taking the lead in helping users to interact with big data and to turn it into actionable data.
Big Data in Action
The FreePint Topic Series: Big Data in Action ran from April to June 2013. Visit the Topic page to find out more and see the links to the published articles. You can still register your interest to receive a free copy of the FreePint Report: Buyer's Guide on Big Data published in June 2013.
We will be revisiting the subject of big data in a new FreePint Topic Series coming later in 2014, focusing on big data and information skills.
- Blog post title: Big Data Techniques Afford Valuable Insights into Government Data
- Link to this page
- View printable version
- Briefing: Public Data in the Context of Big Data
Monday, 5th May 2014
- Privacy - the Tentacles Stretch Ever Further
Friday, 11th April 2014
- Big Data - Talent Gap "Not Where You Think It Is"
Friday, 21st March 2014
- The Internet of Things (IoT) in the Enterprise
Monday, 6th January 2014
- Mini Review: Duedil
Thursday, 23rd May 2013
Five newly recorded webinars on how to complete the activities in our Focus on Value Chain now available.
Express value the way stakeholders do with Value Chain analysis
"I can now sit in a meeting with my stakeholders and articulate the role information plays in their workflows. They understand our contribution to their success." Information Senior Scientist, heavy manufacturing industry
With Jinfo Consulting, you get accountability, fine-tuned guidance and industry-wide perspective.
- Capacity comes from strategy
Wednesday, 15th September 2021
- Tie portfolio value to commercial impact
Monday, 13th September 2021
- Theory, practice, and demonstrating ROI
Thursday, 9th September 2021
- Centre of Excellence (step five - your evolution work plan)
Wednesday, 11th August 2021
- Centre of Excellence (step four - assess your team’s characteristics)
Tuesday, 22nd June 2021
- Centre of Excellence (step three - envision your Known Fors)
Thursday, 10th June 2021
- Community deck - Centre of Excellence - make progress on your evolution, with your peers
Friday, 23rd July 2021
- Community deck - Centre of Excellence - examples of Pillars of Service and Known Fors
Wednesday, 23rd June 2021
- Community deck - scorecards - how information teams are designing and using them
Wednesday, 2nd June 2021
- Investment philosophies for strategic portfolio management (Community) Thursday, 2nd December 2021
- Negotiation role-play for content licensing professionals (Community) Thursday, 7th October 2021
- Teeing up value-based budget conversations for content licensing (Community) Tuesday, 21st September 2021
- Value Chain (step five - putting the Value Chain together) (Webinar) Tuesday, 31st August 2021
- Value Chain (step four - measurement - selecting and collecting data) (Webinar) Monday, 30th August 2021
- Value Chain (step three - your contributions to the Value Chain) (Webinar) Friday, 27th August 2021
Our proven processes, resources and guidance will help your team make the shift from transaction centre to strategic asset.
Designed around the most common challenges and pain points for time- and resource-strapped information teams
A tailored overview of our research and active discussion with your Jinfo analyst.
Measure your starting point to articulate your strengths and set priorities for future improvements. Assessments gauge risk, capacity, value and more.
Read case studies, and start the conversation: