Summary of Focus Group @ Whitireia

• A panel of Assessors concurrently interviewing the Migrant may be advantageous: it mitigates risk of corruptibility, it provides a basis for in-person discussion regarding the competency of the Migrant, and it allows for a more holistic approach to questioning that would not be possible with one-on-one interviews.

• Have the candidate give a write up their skill-set and what they have worked on. This is how we pick the people that will sit on the panel of Assessors.

• Ask generic questions: e.g. What project have you worked on? Then you pick up clues and issues, then you go deeper, then deeper with further questioning

• We could get 10 generic questions from all companies about each skill-set that they look for when hiring. Then keep these in a database which is made available to Assessors.

• Assessors could go through a workshop on what NZQA Level 7 is before they are allowed to interview. NZQA could run the workshops.

• That academic moderation systems require a cross section of work to be moderated each year for each paper within each institution. A similar approach could be applied to Skills Assessments.

Standardized Question Bank Assessment

The pros/cons of a question-based skills assessment test. I recently sat a timed test by a well-known skills assessment service. The test consisted of 54 multiple-choice questions, each question could have up to 3 correct answers, and you had about 2 hours to complete the test.

My research was to assess the possibility of working with this company, to use their tests as an option for assessing skills. Incorporating their question bank and tests into the skills assessment network. Maybe this could work as a preliminary, pre-interview skill checker. Or potentially if it works, then maybe we could use such a service to assess skills instead of a technical interview.

I have experience in software development with Ruby on Rails, although it has been 5 months since I've seriously touched code, I could easily be employed right now as a Rails Dev. I took the ruby-on-rails assessment; I'm pretty sure I failed the test.

One 'pro': minimal human involvement. With a standardized test, a person to interview the migrant is not needed, and reports are generated automatically. The way data is captured through a chunk of multiple choice questions leads to easy analysis.

The other side of the coin is an individual can cheat on the test unless they are monitored. Needing to have a test monitored in person reduces the ability for assessments to be handled remotely. A representative to this company informed me that the only way to ensure the test taker is not cheating would be to have them in a room, monitored and ensuring they are not using their phone or other devices to answer the questions.

I found that most of the questions required me to have technical syntax memorized relating to using the ruby-on-rails framework. In my experience, a good developer is really just a good googler.

There was also something that irked me about sitting another standardized test, particularly because there's been quite a bit of research around the negative effects of standardized testing in school systems. I am unsure if this research would correlate to this application, but with standardization comes the possibility for questions to shared amongst migrants attempting to defraud the immigration system.

It additional takes 6 months to develop an assessment for a technology, meaning that the tests are always going to sit at minimum 6 months behind current standards.

Trademe focus group

The people in the room: 10 senior hiring technical staff and 1 head of HR. 1 person in the room who was a migrant and did not have a bachelors degree, He struggled to get residency.

Big "P" - Policy options

Brought up by one individual: "What if we put a salary cap on devs so from their, their skills do not need to be verified. Arbitrary example of 90k"

Unanimous response by rest of group: "A salary cap would discriminate against small companies because some smaller companies might need skilled individuals but might not be able to pay them top salaries. Likely looking for intermediate."

Alternatively, the purpose of assessing skills is to mitigate risk from Immigration New Zealand's perspective and prevent a person from buying their job offer, an extra salary threshold does not mitigate the risk of a person buying their job offer

Closed vs Open Board

Should this service be performed by a closed group of people? Is it fair to hold companies accountable, instead the people should be accountable?

"The group of people would need to stay fluid"

"The only way to keep individuals on the board part time would be for board members to be part time, and still be working."

Would likely need to be a network of vetted individuals to form the network that represents a board of part-time people who represent the skillset of the industry.

Trust Companies or Trust Individuals

Question Raised: Should we pull people that make up the network from companies, or as individuals. Will companies have biases towards other companies?

Solution: The network as a whole should be individual focused. But seeded from trusted company's hiring staff.

Framework

We talked about different framework models:

• Framework, and skill matrix do not work
• Standardized questions are not ideal, because the can be copied and shared, and they might incorrectly assess a persons skillset.
• SFIA - "Is to broad to be applicable in this situation."
• Other options include code challenges, and technical questions that just shows someones thinking process.
• choosing a champion model - Seems like the best option to trust the accessor

How trademe assess skills

Trademe does open ended take home tasks. And you know in the interview if they can explain why they write the thing that they do.
They also have technical questions that they ask in their technical interview, followed up by a technical challenge that they may or may not collaboratively solve with the interviewer. Usually, 2-3 people are in the room interviewing the candidate. Total of 5 interviews.

tech test is code that does not work, the interviewee will write the unit test and getting the code to pass.

Skills assessment as a COOP

People in the room liked the coop model. And liked the idea of the industry owning the service.

Process

The consensus in the room was that a good process is more important than a framework.

the process needs to be:
* Scaleable - meaning that the ways that assessors are added to the network require minimal effort by the platform
* Not all tech individuals are fit for interviewing. So there should be criteria around who can be an assessor
* The process might want to include inherit ways of testing and verifying the assessor
* The process should be able to work remotely
* ISO standard, Organization that accredited the process.

ways to vet the assessors

• Verify their role in a company (Senior technical or hiring position, chapter leads and delivery managers, do hiring, hold performance reviews, team lead. )
• Do they have experience interviewing before?
• They likely have a range of skills that they can assess, and oppositely a range of skills they can not assess
• Statistically compare assessments to others. Maybe this means not "trusting" assessors until they have proven that their assessments compare with other assessors. eq: have a new assessor perform a 3rd assessment on individuals and to verify the decisions match, before their decisions can be used as the actual judgment of skill.
• Pay assessors to watch interviews and rank the quality of the interview.

Accountable Assessors

Assessors will have to hold accountable for their actions. there are the different option that could work.

• Ability to be an assessor will be revoked if assessments are deemed unfit based on auditing
• Auditors may go through a small training session and sign an agreement to be truthful.

Identify the Champion Framework

After discussing ways that "Identify the Champion" can be implemented. All people in the room came to a consensus that Identify the champion with a stringent process is the best way to conduct assessments and is scalable.

Define tech skills

There may need to be a language to describe what skills people have as a way of connecting people of similar skillset.

Characteristics looked for when hiring

70% technical, 30% soft skills, social, and aptitude to learn, cultural fit, communication, drive, enthusiasm

SFIA

SFIA is far too high level - Indication of employability, not technical capabilities.

Industry representation

Trademe feels like there is not an industry body that currently represents them as a company.

Workshop with comp sci professor of Victoria University

Relationship with IITP and thoughts on SFIA

"Have you talked to Paul Mathews? I'm assuming he said CITP and SFIA"
"My CITP registration is not current. I did it because I am pro professionalism, and being CITP is professional for an academic." "I was grandfathered in by being a fellow of computer society. I'm meant to go through a computer-based assessment, I think SFIA, I tested myself, it asked for things around project management, and I do not think I would pass. I don't think it worth going down to a testing center to do the test. so My registration has lapsed. That gives you an idea of how valuable this is to me."

CITP

CITP has a requirement for 3-year degree and practice for a couple of years, prepare a portfolio and sit exams, so if people are CITP certified, so anyone who is CITP certified with a degree should qualify for residency. But "CITP is too high a bar."

Framework

"A framework will need to change with time." "having a framework that has rankings on different characteristics is more structured that you want."

"Rather than come up with a knowledge framework, you want as brief and general guidelines as possible."

Ideas:

1) 5 years in Industry (not employment) (If you say below 5, then tertiary entities will be upset because it would invalidate the need for education
2) Done 3 separate projects
Then interview would be teasing out this information.

Identify the champion

He suggested we use the framework. "Identify the Champion."
Followed up by sending this link to the resources:
http://scg.unibe.ch/download/champion/

a. Yes
b. Leaning towards yes, but won't be upset if they are turned down.
c. Leaning towards no, but won't be upset if they are accepted.
d. No

A pattern for running assessments to decide which research papers they like.

Process

the process of two people performing interviews then having the assessments compared
"I like that because it has people in industry altruistically giving back to industry."

a person should indicate 1 - 3 technical areas to be linked up with the assessors

Assessors

Who assesses the assessors? Peer review has professional standards, and the person career is over if they do things incorrectly.

Accreditors should sign a code of conduct.
Use data to verify the assessments.
Accreditors will have their assessments audited.

Network

He pointed out that no one has the time to give an assessment. Which is mitigated by accessing many people from many companies, companies contributing for altruistic reasons? Small companies contributing for profit.

Summary

Find out how to vet the assessors, and trust their assessment.

Focus Group with FluxMedia
22 full-time staff, add more people based on project size of 28 contractors 6 months - 18 months. They are the next largest animation studio next to Weta

Problem

Has a big project coming up, literally can not find the staff in NZ to work on these problems. Had to farm the work out to the Philippines. Currently hiring staff from china, malaysia, India.

Would really like to bring them over on a trial first that can be paid, then hire them full time.

The technology that FluxMedia uses is not taught in any school! All staff members are self-taught.

"The skills shortage is global, many NZ animators are poached to Canada or Europe, where there is a less lumpy supply of work."
"Getting staff in from overseas is make or break for our company"

Typical Migrant hired

Staff is hired on their show real alone. Usually, this means that they have 10+ years but also this experience is hard to prove based on the countries that they are coming from.

Industry Bodies

FluxMedia is not represented by any Tech Industry Representative Bodies.
The only reprasentative body is the CDC Group
http://www.scoop.co.nz/stories/BU0807/S00501/cdc-auckland-group-formed.htm

#Framework
Supports a modular light weight framework.

Requests

Keep academics out of this service, their incentive is to send people back to school, which does not reprecent the industry needs.