Sun 10 Apr 2016

Source code/project catalogue (discovery tool)

Brent Wood Public Seen by 374

Source code repositories are good for devs - bit not for managers/policy people looking to make strategic decisions around software purchase or development. Like open data, delivery is easy - stick it on the web in a suitable format :-)

The critical precursor - a discovery capability (not dev oriented) is required. This could be a standalone catalogue of relevant projects providing open source for govt initiatives, or perhaps incorporated in data.govt.nz.

However it is done, a suitable catalogue or discovery portal focussed on projects rather than code should be established & NZGOAL-SE should mandate its use.

This could be established very soon, & easily, & it could be populated with past & current projects.

Such a catalogue should link to the dev & home pages of software projects which have been reused (Silverstripe, Koho, Moodle, Alfresco, etc all spring to mind) as well as the local projects which reused them.


Cam Findlay Sun 10 Apr 2016

Funny enough, this is mentioned in the US FOSS policy under consultation at the moment too. See https://github.com/WhiteHouse/source-code-policy/blob/gh-pages/pages/Implementation.md

US are going to be setting up https://project-open-source.cio.gov/ as a potential discovery and guidance space. Would something like opensource.govt.nz registry or something to that effect be desirable? I note in the NZGOAL-SE there is provision for setting up further guidance notes once the licensing framework is in play.

It would be good if the policy asked as part of the release process to list it on some central discovery space (who should maintain this?). However, the set up of this discovery space would have to happen outside this policy perhaps? I know this is something I've been thinking about for Common Web Platform and reusable FOSS code generated there.

Interesting discussion @brentwood :thumbsup: keen to explore more and determine where this fits.


Rob Elshire Sun 10 Apr 2016

I'm keen to see this included somewhere. It is all well and good to have FOSS licensed software that exists somewhere in a version control system, but it has little value if it can't be found.


Roy Storey Sun 10 Apr 2016

I could see this working as a nz.govt file in the repository of the software in question, which is aggregated by opensource.govt.nz service. Examples for most languages and OS level distributions exist, think CRAN, CPAN, Rubygems, Debian APT, RedHat YUM, Windows Chocolatey, OS X Macports. Most of these have automated bots parsing the metadata files, all of which are subject to pull requests.

What is missing from these is "for managers/policy people looking to make strategic decisions around software purchase or development"

Policy wise: open source projects should include a file in format including adequate information to create an entry at .govt.nz


Cam Findlay Wed 13 Apr 2016

Would it be a workable position for NZGOAL-SE to say something along the lines of "to aid in discovery, released code projects should be submitted in the nominated discovery portal listed in any practical guidance notes"? The guidance notes then would be were we can work out the technical and practical details @roystorey mentions.

I note there is already a section for allowing further guidance notes in NZGOAL-SE draft (Section 7). The guidance notes are a more informal and updatable artefact than the NZGOAL-SE policy itself. Capturing the core principles in NZGOAL-SE and building out further practical guidance notes would help longevity and ongoing usefulness I'd imagine.

I think this might be similar to the thread on the principle of releasing in an open repository that we've had a vote on already. In that case we agreed on a principle and that we could address in specifics in practical guidance notes going forward.


Edward Abraham Wed 13 Apr 2016

I agree that this should be outside of the policy itself. Directories of things need sustained effort to maintain (which means funding). Unless actively curated, they are often stale and so not that useful. I think the focus should be as narrow as possible (choosing and applying an open source licence), with the rest of the ecosystem around that being able to develop as required. Unless there is a funded mechanism, perhaps this shouldn't be included at this stage.


Dave Lane Wed 13 Apr 2016

We could look at repurposing something like the (open source) CSIRO Permanent URL service for data sources (see this instance of it http://registry.it.csiro.au/) ... whatever it is should be self-service registration, so that project coordinators can update the pointer to their projects in a central place themselves. It's all about getting the incentives right :)


Cam Findlay started a proposal Wed 13 Apr 2016

Specific discovery tools should not be included in NZGOAL-SE as this would be out of scope (except to reference them in subsequent guidance notes). Closed Fri 15 Apr 2016

by Cam Findlay Wed 26 Apr 2017

Appears an acceptable and workable way forward on this appears to be to look at specifics in future guidance notes, with a more principled reference in the policy itself. :thumbsup:

There has been discussion here about code discovery tools and their place in NZGOAL-SE. I think we have some agreement that we should keep NZGOAL-SE scope to the licensing aspect and would like to take a quick poll on this to drive further discussion.

Agree - 7
Abstain - 7
Disagree - 7
Block - 7
7 people have voted (17%)

Edward Abraham
Thu 14 Apr 2016

Important there is some guidance of how to do this somewhere. But keep the core goal of getting software openly licensed firmly in focus.


Cam Findlay
Thu 14 Apr 2016


Dave Lane
Thu 14 Apr 2016

I do think it's within the scope to say "the repository, which will be kept up-to-date must be registered with the provided central project directory"... without needing to define that solution.


Roy Storey
Thu 14 Apr 2016

What @davelane said


Rob Elshire
Thu 14 Apr 2016


Tom Clark
Thu 14 Apr 2016


Byron Cochrane
Fri 15 Apr 2016

I concur with Dave Lane. But require entry in an audit-able and publicly accessible authoritative registry (without specifying software) as minimum requirement of publishing. Use similar language as in the versioned repository discussion.


Brent Wood Thu 14 Apr 2016

I think it is critical that DIA (& LINZ) realise that a delivery system without an effective discovery capability is pretty much a waste of effort. Given the lack of progress on geodata.govt.nz & limited support for data.govt.nz that has been apparent for the last few years, I'm far from convinced this is appreciated.

A policy or guideline recommending agencies follow a certain course, with no tool;s or framework to facilitate this, will, at best, result in a plethora of different, probably non-interoperable systems set up by each agency that tries to comply.

Providing guidelines, and a framework with tools enabling compliance is a big step in making it easy to comply with the guidelines, or at least removing significant barriers to adoption.

Much better than "here is what we want you to do - now go & work out how",
is "here is what we want you to do, and here is how you can do it"

NZGOAL-SE is not the place to contain the tools & docs, etc, just as NZGOAL points at CC rather than replicating everything there. It also uses CC to provide examples of how to follow the NZGOAL guiodelines - setting up links to online licences, etc.


Byron Cochrane Thu 14 Apr 2016

I would have to largely concur with Brent on this point. (And I am the guy nominally in charge of geodata.govt.nz!) While I agree with the general point that specific discovery tools need not be named, I do think that at a minimum, any software that falls under this guidance should be registered in a central registry. This register might be less than a full fledged catalogue. What metadata would be included in such a repository could be debated and evolve over time but it must include the uri to the versioned repository in which the software lives. (The maximum approach would be to recommend that any software that falls under this guidance should reside in a yet to be specified central versioned repository.)
The point for this would be that without at least a simple registry, "publishing" your software is not that meaningful. For the agency producing the code, this allow them to "check the box" and say yes, their code is published. For developers inside and outside of government, discovery is at least possible. An audit of activity would be possible against this authoritative list. Addition information could be linked to items in this register.
While it may not be the place of this document to recommend any particular solution to this problem, the need for such should be stated.


Cam Findlay Fri 15 Apr 2016

@brentwood did you have a position on this proposal poll (since you raised the original discussion point)? :smiley:


Danyl Strype Fri 29 Apr 2016

I agree with flagging the need for code to be discoverable in NZ GOAL-SE, and detailing implementation specifics in supplementary notes.

To address implementation, are we talking about a directory for free code software in general, or the subset of free code software used in government? Even if it's the latter, creating a new system specifically for "open government" software seems like an unnecessary duplication of effort, considering there are already a number of existing general directories:
* Free Software Directory: in existence since at least 2002 and maintained by the FSF. Structured as a semantic wiki that anyone can submit new entries or updates/ corrections to, with an editorial group checking submissions for accuracy before approving. Database rights are released under GNU FDL 1.3 (or later). Note: will only list software that can run on a free code OS.
* Open Hub: launched in 2006 as Ohloh, Sold to Geeknet is 2009, who sold it to Black Duck in 2010. Like the FSD, OpenHub is a wiki, but it also features automated tools that search across code repositories. Database rights are released under CC-BY 3.0.
* FreshCode: An attempt to re-implement FreshMeat/ FreeCode(archived). It accepts logins using OpenId, and can accept automated updates from projects, using "JSON-based database exchange feeds" (I presume). Database rights are released under CC-BY-SA (unclear what version).

Maybe the solution is a cross-government project to make sure all relevant software is included (correctly) in one (or all) of these, identified with a general tag (something like "used in government"), and categorized using more specific tags (eg "maps" or "CRM")?


Cam Findlay Sat 30 Apr 2016

Thanks @strypey - an interesting idea, perhaps one reason I could think of for a government open source discovery portal might be around discovery of both proposed and "finished" applications and code, emphasis on proposed (let's also be honest, open source is never truly "finished").

18F in the U.S. have this site for their digital services, gives a good indication as to the stage things are at for a project. I can find out details of who to contact for a project in the discovery stage and perhaps look to collaborate. Something like this might be useful to be checked as a first port of call when thinking about building something new for government (for the technically minded, it uses a Yaml metadata file in the referenced repos to keep dash data up to date, that's been mentioned previously and not something to solve now).

Maybe Agency B finds out that Agency A is in the discovery phase of a new open source GIS mapping application which is pretty close to what they need (through the portal, Agency A has indicated to others their intentions to built such a thing). They should as a matter of course look to work together as opposed to building it in isolation and THEN registering it on a discovery space only to find they built the same things.

I guess the principle to extracted here, regardless of the discovery space being an existing community one or a specific nz public sector one is the "Agencies should do an environment scan of existing open source software discovery portals to ensure they are not building something already built".


Danyl Strype Sat 30 Apr 2016

Absolutely. There's a parallel here to academic research projects, where the first thing you do before you start your research is a literature review, to make sure as much as possible that you're building on existing research, not duplicating it. In fact, this analogy might be a useful way of explaining the relevant practical advice in NZ GOAL-SE. A classic real world example for software would be CMS. It would be a crazy use of public funds for a government department to build their own CMS from scratch (or lease a hosted proprietary one), rather than building an special features they need as modules or plug-ins for an existing free code CMS like WordPress or Drupal (or Totara in @donchristie 's Customs example). Indymedia learned this the hard way (as mentioned elsewhere).


Brent Wood Wed 24 Aug 2016

Given code is data of a specific type - should data.govt.nz support software - as part of the overall capability if not the same catalogue/toolset. It seems they pretty much already have a mandate....


Cam Findlay Wed 14 Sep 2016

I'm also keeping an eye on this https://code.gov/ :eye: