ITCC June Education Session - Panel Discussion on accreditation

Summary

The video script features a panel discussion on the topic of accreditation in certification programs. Panelists from various organizations, including Microsoft, the Linux Foundation, and CompTIA, share their experiences and insights on the value and challenges of accreditation, the importance of maintaining standards, and the potential for innovation within the industry. The conversation highlights the balance between adhering to rigorous standards and pushing the envelope to improve and evolve certification practices.

Takeaways

• 📅 The ITCC meeting will be held at SAS headquarters in Cary, North Carolina on November 6-7, with registration opening on August 1.
• 🎉 The panel discussion featured a range of perspectives on accreditation, including representatives from Microsoft, the Linux Foundation, Criterion, Alpine Testing, and CompTIA.
• 🛂 The importance of accreditation lies in providing an independent validation of the certification process, thus lending credibility to the assertion of an individual's professional competence.
• 🚫 The 17024 standard, which accreditation is based on, prohibits a training program from being too closely tied to a certification program to maintain independence and avoid mandating training as a prerequisite for certification.
• 🔄 There is a shift towards more flexibility in accreditation standards, allowing organizations to innovate while still meeting the established criteria.
• 💡 The panel suggested that organizations should be prepared to defend their methodologies and demonstrate how they meet accreditation standards, even if they differ from traditional approaches.
• 📈 Microsoft's experience with accreditation highlighted the rigorous standards and the potential for using these as a guide to improve program quality, even without current accreditation.
• 📊 Data and evidence play a crucial role in demonstrating the effectiveness and validity of new or innovative approaches to certification and exam development.
• 🛑 The cost and effort of maintaining accreditation can be burdensome, and organizations must weigh the benefits against these demands, as seen in Microsoft's decision to not pursue accreditation.
• 🤝 The panel encouraged open communication and willingness to challenge the status quo in accreditation, pushing for innovation while adhering to high standards.
• 🔑 Accurate documentation and clear policies are essential for organizations seeking accreditation or looking to innovate within the certification landscape.

Q & A

What is the main topic of the discussion in the video script?

The main topic of the discussion is accreditation in the context of certification programs, with a focus on the experiences and perspectives of various organizations such as the Linux Foundation, CompTIA, and Microsoft.

Why is accreditation important for certification bodies?

Accreditation is important as it provides an independent validation of the certification process, lending additional credibility to the assertion that an individual or organization has met a minimum threshold of professional competence.

What does the 17024 standard refer to in the context of this discussion?

The 17024 standard, now known as ANAB, refers to the accreditation criteria for organizations that certify personnel. It ensures that the certification process is rigorous, fair, and valid.

Why did the Linux Foundation initially decide against pursuing accreditation?

The Linux Foundation initially decided against accreditation because their clients were not asking for it and were not familiar with what it entailed, making it hard to justify the process and documentation required for accreditation.

What are some of the challenges organizations face when seeking accreditation?

Some challenges include conducting a job task analysis with a broad and diverse representation, maintaining thorough documentation, adhering to the standard's requirements, and demonstrating ongoing compliance with the standard.

How does CompTIA benefit from being accredited?

CompTIA benefits from accreditation by meeting a requirement of certain organizations for doing business, providing internal discipline to their operations, and offering a solid onboarding experience for new staff members.

What was Microsoft's reason for initially pursuing accreditation?

Microsoft initially pursued accreditation because of a directive from the Department of Defense (DoD) that required accredited certifications to be listed for specific roles in the military, primarily related to security.

Why did Microsoft decide to drop their accreditation?

Microsoft decided to drop their accreditation due to the lack of return on investment they were seeing and the burdensome annual renewal requirements, which were not justified by the benefits they were receiving.

What is the role of a consultant when an organization is seeking accreditation?

A consultant can help guide an organization through the accreditation process, ensuring that they meet the necessary standards and documentation requirements. However, it's important that the consultant is willing to push the envelope and support the organization's innovative approaches.

How can an organization demonstrate its commitment to quality without accreditation?

An organization can demonstrate its commitment to quality by adhering to industry best practices and standards, such as those outlined in the ISO or NCCA standards, and by maintaining rigorous internal processes and documentation.

What is the potential drawback of the accreditation process for innovative organizations?

The potential drawback is that the accreditation process may stifle innovation by imposing strict rules and standards that can be difficult for organizations to adhere to while also pushing the boundaries of what's acceptable in the industry.

Outlines

00:00

📝 Introduction to ITCC Meeting and Agenda

The speaker opens the ITCC meeting by thanking attendees and introducing the format, which will include a discussion on accreditation. The panel consists of experts from various organizations including Microsoft, the Linux Foundation, and CompTIA. The speaker hints at changes and asks attendees to stay tuned for announcements and a poll at the end of the session. The focus is on the importance of accreditation in the certification process and maintaining a separation between certification and training development.

05:01

🏛 Importance of Accreditation and Misunderstandings Clarified

Clyde Superat from the Linux Foundation explains the significance of accreditation in certifying individuals' professional competence. He discusses the role of accreditation in providing credibility to certification bodies and the importance of maintaining a separation between certification and training to ensure independence and avoid conflicts of interest. The standard ANSI/ISO/IEC 17024 is mentioned as a benchmark for accreditation, and common misunderstandings about the inability to have both training and certification within the same organization are addressed.

10:03

🔄 Linux Foundation's Experience with Accreditation

Clyde shares the Linux Foundation's journey with accreditation, including their initial decision not to pursue it due to a lack of client demand and the costs associated with the process. He outlines the rigorous nature of accreditation, the need for client education, and the internal benefits of maintaining high standards. The discussion also covers the challenges of job task analysis and the necessity of broad community representation.

15:05

📈 CompTIA's Perspective on the Value of Accreditation

Carl Bo Bowman from CompTIA discusses the organization's experience with accreditation since 2010, highlighting the benefits of discipline and documentation it brings to their operations. He acknowledges the effort required for annual renewals and the involvement of various teams within the company. Bowman emphasizes the importance of accreditation for revenue opportunities and maintaining quality standards.

20:06

💡 Microsoft's Decision on Accreditation and Innovation

Liberty Munson from Microsoft shares the company's experience with accreditation, which was pursued to meet specific military requirements but later dropped due to a lack of return on investment. She discusses the challenges of maintaining accreditation, especially for a small team, and the potential stifling effect of standards on innovation. Munson raises concerns about the compatibility of accreditation standards with new approaches in exam development and accommodations.

25:06

🤔 The Role of Consultants and Assessors in Accreditation

The panelists discuss the role of consultants in helping organizations navigate the accreditation process, emphasizing the need to find consultants who are willing to challenge the status quo and support the organization's unique approach. They also touch on the subjectivity of assessors and the importance of having data to back up an organization's methodology and compliance with standards.

30:09

🛠 The Challenge of Balancing Standards and Innovation

The conversation continues with a focus on the challenges of adhering to accreditation standards while also innovating in certification programs. Panelists share their insights on the need for organizations to document their processes and stand by their methodologies, even when they differ from traditional approaches. The importance of data in supporting these methodologies is highlighted, as is the potential for accrediting bodies to be more flexible in their interpretation of standards.

35:10

📚 Documentation and Governance in Certification Programs

Karina Owens and Zach Owen discuss the importance of having clear documentation and governance processes in place when pushing the envelope with certification programs. They stress the need for organizations to be able to demonstrate their adherence to standards and to have a clear rationale for any innovative approaches they take. The panelists also share advice on dealing with accreditation reviewers and the importance of being prepared to defend one's methods.

40:10

🚀 Embracing Accreditation Standards for Program Improvement

Liberty Munson suggests that even without pursuing accreditation, organizations can benefit from reviewing the standards as a guide for improving their certification programs. She emphasizes the value of the standards in setting a rigorous path for program development and quality assurance, regardless of whether an organization chooses to go through the accreditation process.

🎉 Conclusion and Upcoming ITCC Meeting Announcement

The session concludes with a summary of the key points discussed and an announcement of the upcoming ITCC meeting to be held at SAS headquarters in Cary, North Carolina. The panelists encourage attendees to provide feedback through a poll to help shape future presentations. The meeting is set for November 6-7, with registrations opening on August 1, and the organizers look forward to seeing attendees there.

Keywords

💡Accreditation

Accreditation refers to the formal recognition of an organization's certification program by an independent body, ensuring that the program meets specific quality standards. In the video, accreditation is a central theme, with various panelists discussing its importance in lending credibility to certification programs and the process involved in achieving it.

💡Certification

Certification is a credential that individuals obtain to demonstrate their professional competence in a specific area. The video discusses the role of certification in the tech industry and how accreditation impacts its validity and recognition, as seen when Clyde talks about the assertion of an individual's capabilities through certification.

💡Standard 17024

ISO/IEC 17024 is an international standard that specifies requirements for the bodies providing certification of persons. In the context of the video, it is the standard against which organizations are accredited, ensuring that their certification processes are rigorous and credible, as Clyde explains when discussing the importance of accreditation.

💡Job Task Analysis

Job Task Analysis is a process used to identify the tasks and knowledge required for a specific job role. In the script, Clyde mentions this process as a requirement under the 17024 standard, emphasizing its importance in determining the scope of certification exams to ensure they are relevant and comprehensive.

💡Firewall

In the context of certification and training, a 'firewall' refers to the separation between the development of certification exams and the training programs that prepare individuals for those exams. The video discusses the need for this separation to maintain the independence and credibility of the certification process, as explained by Clyde.

💡Training Blueprint

A training blueprint is a document that outlines the scope and objectives of a training program. The video emphasizes that training organizations should base their programs on publicly available blueprints, without access to the private exam blueprint, to ensure fairness and maintain the integrity of the certification process.

💡Exam Development

Exam development refers to the process of creating certification exams, including defining the scope, writing questions, and setting standards. In the video, Carl and Clyde discuss the importance of separating exam development from training to avoid conflicts of interest and ensure the validity of certification exams.

💡Remote Proctoring

Remote proctoring is a method of monitoring exams online rather than in a physical testing center. The video discusses how Microsoft was the first to adopt remote proctoring and the challenges it faced with accreditation standards, highlighting the need for innovation in certification processes.

💡Innovation

Innovation in the context of the video refers to the development of new methods and technologies to improve certification processes, such as remote proctoring and accessible exam experiences. Liberty from Microsoft expresses concern that accreditation standards may stifle innovation in the certification industry.

💡ROI (Return on Investment)

ROI is a measure used to evaluate the efficiency of an investment. In the video, panelists discuss the costs and benefits of accreditation, with Liberty questioning the ROI of accreditation for Microsoft given the challenges of maintaining it while pursuing innovation.

💡Psychometrics

Psychometrics is the field of study concerned with the measurement and analysis of human abilities and traits. In the video, the discussion around exam development and accreditation touches on psychometric principles, such as the need for rigorous exam development processes that can be defended to accreditation bodies.

Highlights

Introduction of the ITCC meeting and changes in the format with a focus on suspense and engagement for upcoming announcements.

Excitement about the panel consisting of experts from Microsoft, Linux Foundation, Criterion, Alpine Testing, and CompTIA.

Discussion on the importance of accreditation in certifying individual capabilities and its role in providing credibility.

Clarification on the separation between certification development and training development as per the 17024 standard.

The challenge of maintaining independence in certification processes to avoid conflicts of interest.

Linux Foundation's initial decision against accreditation due to lack of client demand and the costs involved.

CompTIA's experience with accreditation since 2010 and the benefits of discipline and documentation in exam development.

The effort required for annual renewal of accreditation and the involvement of various teams within an organization.

Microsoft's journey with accreditation, the initial push for it, and the eventual decision to drop it due to lack of ROI.

Concerns about accreditation standards stifling innovation in exam development and the challenges of remote proctoring.

The importance of having data to support new methodologies in exam development and the role of evidence in accreditation discussions.

The evolving flexibility of accreditation bodies and the need for organizations to defend their innovative approaches.

Advice for organizations considering accreditation, including the importance of selecting a consultant who supports innovation.

The value of accreditation in differentiating a high-quality certification program from less rigorous ones.

The upcoming ITCC meeting at SAS headquarters and the importance of marking calendars for the event.

Transcripts

thank you for everyone uh for joining

today um we'd like to start out with a

quick statement about uh itcc

meetings um and also um we're we're

going to be switching things up at a

little bit today um so um we're going to

um am I talking too fast or the slides

too slow I'm not quite sure what's going

on SLS are too slow so sorry about that

Graham I can talk slower um anyway so uh

we're going to be switching things up a

little bit today uh so we're going to

keep you in suspense uh so please do

stay tuned uh for

announcements um at the end alongside

the uh the poll that we like to have um

after the

sessions so today do you know what I'm

smiling because it's like uh Hollywood

aists the panel that we've got today so

really excited um our education session

today will include a discussion on uh

accreditation accreditation the panelist

will include Liberty mson from Microsoft

Clyde superat from the Linux Foundation

Zack Owen from Criterion we have also

have Karina Owens from Alpine testing

and Carl Bo Bowman from CompTIA so uh

welcome to you all uh thank you so much

for being here today our a-listers and

um without further Ado I'll pass over to

Christen uh to modate the

panel all right well thank you very much

and um I realized as I was just um

getting ready to go on camera that um I

am wearing one of my favorite its dog

t-shirts so even though it's not cat

shirt Tuesday it's dog shirt Thursday

over here anyway so no I hope no offense

taken to any other venders but sorry

this is a comfy tea so with that let's

go ahead and get started um I wanted to

first start out um for anyone um who is

new to to sort of the idea of

accreditation and whatnot first going to

go ahead and start out and Clyde is

going to talk about kind of the need why

why is there this importance of having

this sort of this this wall between um

certification development training

development um and so on from his

perspective of the of the work he does

outside of Linux foundation so

Clyde let me unmute thanks um Kristen um

so I'll give share some commentary it's

largely informed by uh my rule the past

I think four maybe five years serving on

the accreditation committee for an for

17024 uh which is now called anab not an

um but that's the body that handles

accrediting organizations for certific

person certification which is the 7024

standard um I guess maybe just a word

first on why accreditation at all um and

it really has to do with you know when

we certify individuals through a high

stakes process we are making a assertion

about that person's capabilities right

we're asserting that they are they've

met a met a minimum threshold of

professional

competence uh and the accreditation

process is basically a way of lending

additional credibility to the assertion

that an individual organization makes

right so you can imagine you know an an

org has a CT and somebody that's not

familiar with either the CT or the org

say well why should I trust them like

what what uh evidence do I have that

they've done a good job collecting

defining a good scope what evidence do I

have that theyve done a good job writing

an exam as representative and so on and

so forth and so you know what

accreditation does is provide a level of

independent validation on the process of

yous to come up with the credential that

you created and therefore the the

assertion that you are making about the

competence of the person who passed your

exam and there's a lot of use cases

where there's uh demand for that level

of independent um support for the for

the asss that that you

make uh in particular where most

training organizations or most uh Tech

organizations uh run a ground early is

this idea that that uh can you have

training and

certification both in your org and this

is one of these where there's been a

little bit of

um uh misunderstanding I guess is the

best way to put it uh and there's a

there's a uh belief that taken hold that

that the under the 17024 standard you

cannot have both uh which is not true

what the standard tries to do is to say

uh you cannot have a uh a training

program that is attached to the hip to

your certification program for primarily

two reasons one if your if your training

is basically a rubber stamp version of

the exam

um and you know uh color by members exam

prep what you've essentially done is uh

or pretty much mandated that somebody

take your pay for your training in order

to take the exam because uh it's clear

to the market that this is the approved

way in which that you in which you in

which you can prepare for and pass the

exam and that's not allowed right there

a as a certification

organization uh they don't want uh

organizations uh having a uh a require

what is basically a requirement to take

the training in order to take and pass

the

um the

certification and then the the second

thing they want which is the sort of

flip side of that is uh that there

should be a healthy ecosystem of ways in

which people prepare and it's very

difficult to get a vendor ecosystem

engaged and wanting to prepare people

for an exam if they would on the street

is the organization has made it so that

you basically can't uh compete uh so

what does that mean for organizations is

think about it this way your training or

should be acting in the same way that

they would act if they were a third

party organization and so what that

tends to mean in practice is things like

uh they should be going off the publicly

available blueprint uh that's made

available to everybody right as to

here's what here's the scope and the the

the statement of

expectations uh the what and and by

implication that means they should not

have access to the private exam

blueprint that says okay there's 40

things on here here's the 22 things

we're going to test right and so there

needs to be a clean break between the

the public you know version of the

blueprint that gets publicly made

available and that's what the training

organization can utilize because in that

sense they're no different than any

other third party trainer coming up and

trying to build a training program and

then the second major no no is you can't

have people on the training side of your

house participate in the actual exam

development process for questions uh for

the same reason right so that they you

know because then they get a specific

insight into what

um what exactly is on the exam and how

it's being framed and so the firewall is

ready

uh does the training team have access to

specifics of what's on an actual exam

form or not and uh making making sure

that the people who are writing your

training program were not also actively

developing the content so they can

participate in the GTA along with a

broad Group which which required to have

multiple people providing input on the

development of the scope uh but what

they can't do what you can't do is have

people on your training team then

participate in developing actual items

and go on the exam so hopefully that

creates a little bit more clarity as to

as to what the so-called prohibition on

training is it's a very practically

minded um call to maintain Independence

specifically when it comes to

preparation for the

exam thank you very much for that Clyde

Laura did you have a

question okay so look like audio came up

all right so um the way we are going to

um handle the this call is we've really

got three different groups here right so

we have a Linux Foundation which is

actually looking at accreditation uh we

have CompTIA who is actively

participating in accreditation and then

we have Microsoft who tried

accreditation decided it wasn't for them

and went the other direction um and then

um we will have um Karina and Zach from

the vendor side who can kind of talk

about what it is that they're seeing um

with their customers and so on so that's

kind of the progression that we're going

to go ahead and take um and I've got

some questions for each one of them so

um Clyde let's go ahead and start with

you um you mentioned in the preall that

you guys had Linux Foundation had looked

at doing accreditation before decided no

but you're now back at it um do you want

to tell us

why yeah we

uh the short version is the clients

didn't weren't asking for it and didn't

know what it was and so it was just hard

to justify all the process and

documentation that you have to put in

place in order to be accredited so we

operate our program largely in alignment

with the 17024 standard which is a

standard that you get accredited against

um but there's a there's just there's a

lot of process right ANC uh ISO

accreditation is a process accreditation

right they don't uh they're looking at

are you adhering to the process rather

than your specific

um outcomes uh or specific practices you

it's on your responsibility as a as a CB

is to demonstrate how you adhere to the

requirements of the standard and so you

know we like the rigor of it but we

started talking to the clients and the

clients were just like we don't know

what that is uh and I think for that's

really the business case decision is uh

you know it's nice to have I think it

gives a team a good sense of of

confidence that they're doing things the

right way but there is a that's a cost

right there's a cost to getting the

accreditation there's a cost to

maintaining the accreditation there's

annual

recertification uh checks that you have

to do and then there's a re

accreditation process every think it's

four years uh we do have a major client

that is uh on a new program that's

asking again which is why we're back to

thinking about it but you know

fundamentally it came down to could we

justify the cost of maintaining the

formal accreditation and the answer

before was no and the answer now is okay

if we get this big project then it would

be um a worthwhile investment to

make so um what are as you look at going

ahead and um moving into accreditation

what are some of the initial

accreditation challenges or

processes um that you're looking

at so I think it depends a little bit on

what your starting point is right so

we've operated a process at the LF which

has been

largely intended to be compliant with

the 17024 standard um where I see

organizations most frequently have

difficulty against a standard is uh the

way in which you go about doing your job

task analysis to come up with the scope

of the exam you know what you're

required to do under 17024 is

demonstrate that you've included

feedback from a broadc cross section and

ADV diverse cross-section of your

community your user community and that

often can be a challenge for

organizations that develop their

organizations with a very um targeted

sort of primarily in uh

in-house um expertise right and so

having a going outside to the client

base to the user base making sure you

get representation from in different

industry sectors uh to participate in

the in the job task analysis committee

is often one are the things that

organizations sort of um maybe aren't

doing when they first approach sort of

getting the standard done uh and the

second big one is documentation right

there's a lot of requirement for

documentation throughout the standard

right and so it gets down to the level

of like every document has to have like

a footer that says when last it was

revised and so there's a significant

amount of um administrative capacity

that you have to devote towards staying

in compli lents like do you have CER

statements for um confidentiality for

every single person that's working on

the exam there's a whole checklist of

things um because it's a process

certification that you need to stay on

top of and demonstrate that you met the

standard but also your process for

ensuring that you continue to meet the

standard and so those that that those

are the areas where I think most

organizations when they first come to it

find H those are not things I'm doing

today

uh how am I going to go about making

that happen and how many people am I

going to need you know who's going to do

that that

work great so Carl I'm gonna jump over

to you CompTIA um and my first question

to you um and I apologize if there are

any questions I ask you that you don't

have the answer to because you weren't

involved in that part of it um but how

long has comp have been

accredited sure uh thanks kisten

um we've been accredited I believe since

2010 2011 thereabouts uh that's when we

uh originally uh stepped into this and

it was a significant um undertaking as

as Clyde kind of alluded to uh but we've

been we've maintained our accreditation

since then and I think uh obviously over

the years of maintaining that level of

discipline and documentation I don't

want to say it's become easier but it's

become more of a permanent fixture in

the operation and there's a lot of

expectations around doing things certain

ways to maintain our cord

what value have you seen by um actually

being accredited and staying

accredited um I think there's a couple

things so one and one of the reasons

that we pursued accreditation kind of

alluding to what clad was U mentioning

is we had a a series of different uh

organizations that uh were going to

require uh certification bodies to be

accredited in order to engage in

business and so we saw opportunity for

uh basically Revenue opportunity

essentially and so that was one of the

main drivers there and I think that you

know over the years that's something

that's continuously Revisited um so

that's I think really the impetus behind

it if you will from my perspective

though managing the team that that

develops and maintains our certification

exams I think the other benefit

internally is that it really does bring

a lot of uh discipline to everything

that we do um across the board and

having to document things yes certainly

represents additional work uh and even

some potentially some additional

headcount depending on what you're

trying to accomplish uh in the end of

the day being able to refer back to that

documentation having the documentation

readily available to onboard new staff

members provides a really smooth

onboarding experience and training um it

really is frankly I would argue a good

discipline to have despite whether

you're accredited or not um if you

really want to run run a solid

highquality operation where there

there's a lot of focus in on every

single piece of the operation um it's

definitely a a high standard to work

towards um so I think that's kind of

like a dual answer one is the

organization uh you know kind of Revenue

opportunity and the other is just the

internal I think uh quality standards

that it sets for teams that have to you

know maintain it okay and then how much

effort goes into the um annual renewal

renewal requirements

it's it's a lift uh we have one staff

member I would say

dedicates uh that is really kind of

focused in this area and maintains all

her documentation in the in the

management system and when that time of

year comes she spends easily goes into

like this ansy mode and at least

part-time um you know 50% of our time or

more is dedicated to that and it's it's

at least probably two months or more of

prep uh and So that obviously she man is

uh within our team everything that's

required as well as you know HR finance

and a couple other teams that are also

touched by the standard so um yeah it's

it's a it's a lift I'm not gonna not

gonna

lie okay do you have any lessons learn

or words of wisdom that you would pass

on to others that are maybe like the

Clyde just getting started with Linux

Foundation or or someone else um yeah I

mean you

know it's certainly going to require you

to rethink a lot of things that maybe

you've done a certain way for a long

time and you're like well we're going to

have to do this a different way moving

forward in order to meet the standard or

you might look at it and go we're going

to have to bring on a consultant

part-time to help with this uh every

year to kind of you know help us get

through this this additional lift so

that it doesn't burden the rest of our

team or um the other piece is really

kind of educating other internal staff

members that aren't part of the

development team that might again you

know uh there C pieces that might impact

HR or finance and and spending the time

to actually sit on with those executive

leaders and say look you know we're

going to do this we need you to be on

board with this we need your support we

need you to give us the time and uh and

resources from your team members to be

able to you know check all the boxes if

that makes sense and again willingness

to change right uh you really have to be

willing to think differently and think

from a different

perspective sounds good thank you um

Liberty

um so from a Microsoft perspective you

all went through the process to get

accredited you want to tell us kind of

do I even haveing Clyde Carl talk about

how much work this is so we actually did

it because uh I actually was hired into

micro this is how long ago this has been

I was actually hired into Microsoft to

help us get accredited um and the uh I

was hired at Microsoft in 2006 and we

didn't get accredited until 2008 so

that's just how much work it was now I

came into a mess at Microsoft um we

didn't have anything documented so I had

to create all that

documentation um and but the reason we

did it was because at the time the dod

had a directive 55 55,000 something and

one it was some directive where only

accredited certifications could be

listed and it was needed to uh those

certifications were needed for very

specific roles in the military if I

remember correctly it was primarily

related to security and we wanted our uh

certifications uh a few of our

certifications to be listed in in that

directive so that it could they could be

used to help um the military people in

those roles could use Microsoft right so

we saw it as a potential Revenue 8570

thank you um I've just thrown numbers

out at this point uh so that's why we

did it and so I we didn't it didn't keep

it very long um because we did not see

any return on investment and we weren't

seeing that many more people in the

military choosing a mic to take a

Microsoft certification because it was

part of that directive and the annual

requirement for Renewal was extremely

burdensome now let we also have to

remember that even at that time

Microsoft's exam development team was

pretty small I don't it may have been 15

people but there's always only been one

of me and that was was it was a lot of

work um when I have to manage the entire

psychometric elements of all of our

content um plus then add-on all the

requirements to go around with this

annual renewal so I think uh so we ended

up dropping it because we didn't see any

return on investment but we have been

asked to revisit it even in the recent

past and right now and now the challenge

is more related to the standards not

allowing for Innovation right so if

there are things where I want to

experiment with especially around some

of the stuff that I'm doing related to

accommodations and accessibility I have

a hard time believing that we wouldn't

be challenged for some of those

decisions under the the strict rule of

what those standards say I also don't

think our C renewal um process would fly

and so the reason we have been resistant

on even revisiting this is that in my

opinion these standards stifle our

ability to innovate in places we need to

innovate interesting

so I'd be curious to know and I don't

know uh if anyone here knows but from a

generative AI perspective because that

seems to be something that comes up on

every single one of our calls lately um

how would ancy look at the use of AI in

exam development at

all we talk about stifling Innovation so

I can speak to that a little bit Kristen

um so the 17024 standard is under review

right now right so the standard itself

gets reviewed every eight years or

whatever uh and they they are grappling

with that issue of um what do you have

to say about how and when it's

appropriate to use and it's possible

they might end up saying look at the end

of the day the organization signs off on

the standards and the processes and so

how much do we really care about how

that content came to be so long as you

have the panel of experts with

um experience across industry across

geography you know exp you know

experience levels Etc um and can you

still meet the standard in terms of the

rigor with which they were developed so

there's a whole debate about kind of you

know we've always said it was a process

certification and we didn't really care

about the methods so long as you could

demonstrate that the method meets the

standard and so there might be some of

that happening um

and the same is true at the Liberties

point I think the organization has

gotten a lot better in the past three or

four years about being flexible um with

the fact that there could be other ways

to meet the standard um Co you

know before Co it was a massive lift to

try to get them to approve remote

proctoring because that was one of the

things that was unknown know right it

was like the the whole standard was

written with the assumption that you

would Proctor in person and I think

having to you know every program had to

go to to Virtual and and they had to

sort of pivot ever since then I think

the assessors have been more open to

things that would have been taboo um in

the past and so I guess um to your point

I I would actually encourage

organizations to stick to their guns be

willing to explain how the way you're

doing it meet the standard even if it's

not what they typically would have

accepted which is you know you have to

be you know recertified on an exam or

you have to do a a checklist of things

to demonstrate continue I think there's

more openness now to organizations being

able to articulate more forcefully this

is why we believe this meets the

standard and not have it be met with

this idea of nope that's not how you do

it you have to

do

X I think that's a good call out though

because what really should be happening

um I think so we have two crediting

bodies right so aab is ACC crediting

against the ISO standards and then

there's also NCCA who tends to be a

little bit more prescriptive than um the

ISO standards are for sure um but I

think to Liberty's point is it's a what

should be happening is exactly what

Clyde is saying when you present your

application as a psychometrician as the

organization you should be able to

defend the methodology that you have

chosen and they should be able to say

does the evidence that they've collected

and they've provided me adhere to this

standard I think what happens sometimes

and can create some hesitancy is you you

get assigned an assessor and so that

assessor makes the decision as to

whether or not you are adhering to those

standards and so as Liberty and Carl and

Clyde have all kind of indicated it is a

very involved process that doesn't just

adhere to how are you conducting your

psychometrics but there is a whole

another layer of documentation and

things that you have to put in place and

I think organizations that I've worked

with in the past and they kind of run

the gamut so some are accredited some

have some exams accredited but not their

entire Suite of program

or exams excuse me and others have just

chosen not to go that route or have gone

that route and then decided not to and a

lot of the reason that people get a

little bit nervous about going back into

it or moving in that direction is what

if I get an assessor who doesn't agree

with my arguments and I put in all of

this work to get there and then they

tell me no um and the costs so

everything that's kind of involved in

that but I would wholeheartedly agree

with Clyde that we need more

organizations pushing and saying this is

the standard this is is how I have

interpreted the standard and this is why

I feel like I can move forward and are

adhering not only to the iso standard or

the NCCA standard but I've built my

program around the standards for

educational and psychological testing

and here's the evidence to show that it

makes sense to move in this direction I

just want to Echo that uh what Karina

said is data speaks uh and I think the

proof is in the data as well um and

we've had without disclosing anything

confidential here um several situations

where we've had a bit of an argument in

back and forth and said look we have the

data you can't tell us we're wrong the

data is based on all the psychometric

industry best standards it it says that

everything we're doing is valid the

standard doesn't say otherwise it meets

the standard it is what it is whether

the assessor likes it or not um and

we've had those conversations and I

think Clyde to your point we've seen

that as well that

um you know there there's a bit of a I

don't want to say bias but maybe

perceived bias uh for remote proctoring

and when we show data that says our test

center operation and remot proactor

operation are equivalent and we have all

of the data and psychometric reports to

prove

that it then it is what it is there's

there's no way to kind of argue that the

flip side of that coin is and again

setting the side of the NC standard in

my mind if we were running our operation

and found that there was a discrepancy

and a difference in the outcomes and the

validity of the outcomes between those

two we we would have a problem

regardless and we would have to face the

facts and deal with that right so I

think the anoor is coming through it

just adds a layer of kind of um

accountability for CompTIA right but in

the end of the day you know if we're

going to be accountable to ourselves we

still have to do the due diligence to to

seek out that proof If you will that

evidence and take a hard look at it

whether there's an ANC auditor sitting

there or

not which is kind of where Microsoft has

landed right is I hold a pretty high

standard around everything that we do

but I don't have to deal with the cost

of both in time and resources to

maintain the

accreditation I just want to go back to

something Karina said uh because I think

it is a challenge a lot of organizations

when they do it for the first time hire

a consultant to help them and the

Consultants feel like success means get

the client accredited so I would if

you're looking at doing that I would say

make sure you interview the consultant

for their willingness to push the

envelope because a lot of them will be

will try to put you on this very sort of

prescriptive path of like I know we can

get it through if you would just do it

this way so don't be you if you're going

to go to the consultant which most

organizations do the first time around

unless you have a liberty in

house really push them right make sure

they're willing to take you where you

want want to go because otherwise their

def their default is get you certifi get

you accredited and I think there's a

little bit of a dynamic there where

sometimes the Consultants are staring

people to do it yesterday's way because

they're confident that that way will be

approved so Zach I want to just come

over to you I uh and see what you have

to add what you're

seeing um just your thoughts here I was

actually going to chime in when Clyde

said that actually talk to your

Consultants before you uh you hire them

so that you know that they're okay and

and for I like to say picking a fight

with the accreditors uh but uh basically

just going over the process that you

want to go through them identifying

points that might run contrary to the

standards but still getting you in the

same direction and the same ballpark as

you want to go uh rather than as Clyde

said doing it yesterday's way uh through

a prescribed path that only meets that

doesn't meet your needs effectively and

I and I and along with Carl and probably

Karina as well have had those

conversations with the accreditation

reviewers that says hey you know this is

what we did here's the documentation

here it is on this page of the technical

report or whatever other documentation

or policy we've provided that says we're

doing it up to standards and you're

trying to prescribe me a different way

of doing that we don't want to do that

way that meets the standards but so does

our way and why are you why are you

doing that

so yeah it definitely comes down to your

willingness to actually have less

aversion to conflict in some of these

cases because I think a lot of

organizations fall into that trap of oh

the the reviewer said we can't do it

that way so we have to change not stick

into your guns and actually showing the

evidence that your process is equally

valid all right so um so then I guess my

next question and and I'll open this up

to to all the panelists um so Liberty

had mentioned part of what she was

concerned about was going ahead

and we we know that from a certification

perspective we have to stay Innovative

if we want to keep people interested in

certification we can't stay where we're

at you know pure multiple choice

questions and and whatever we know we've

got to we've got to push the envelope

right and so what I think I'm hearing um

and feel free to expound what I think

I'm hearing is that have a good

consultant or not but have somebody

where you're not afraid to go in and say

this is why we're doing it here's the

data behind it and so on and you can be

successful um so I'll just open that up

to anyone else who wants to add on that

or so um I'm happy to hear it sounds

like there's more flexibility around the

interpretation of the standards I will

tell you Microsoft was the very first it

certification company that went to

remote proctoring and we would like that

was when we were going to be out right

because at the time there was no way

that was ever going to fly that was

really like there it was very much

against the standard as Clyde alluded to

we are also the very first program

that's really tried to be a more

accessible experience and so one of the

things that I have done is I've looked

at all of the security requirements

we've had in place and I've tried to

reduce i' essentially remove the ones

where I think the risk is really low I

have a hard time believing that that is

going to be acceptable under the

standard some of the things that I some

of the decisions I've made on to eror on

the side of the candidate versus on the

side of security in my attempt to be

more inclusive and accessible so I while

I'm happy to hear it as Microsoft keeps

trying to push the boundaries of what's

acceptable in the industry I just have I

I'm gonna just continue to have a hard

time believing that it's worth my time

and effort to go even have a

conversation with the creditors to try

to get us back through that process

but by now you have the data Liberty

that says that the the outcomes in terms

of things like the pass rate and uh

achievement inflation over time were

they materially

different because that's the that's the

argument right it's like look security

isn't for security sake Security is to

protect Integrity of the exam so if

you've done it this way and here's six

months of data that says the average

passing score the time on am the P

values but you know if it's not

different then what do you guarding

against because it's the same

result

I it's a it's a fight worth picking to

cost

point I agree 100% the data speaks for

itself um so I I agree with you Liberty

like there has to be the ROI piece right

if there's no Roi piece I mean

I there's I can't try and convince you

otherwise but um but from

a um an auditor perspective uh when you

have the data all laid out on the

table you can't argue with the data and

we've seen that time and time

again Karina and Zach anything to add

from what you've seen with your

customers who are going through

accreditation and pushing

envelopes uh not so much pushing

envelopes but if you are pushing an

envelope you definitely need to have

what you're doing documented in in

policies procedures your governance

processes and be doing what you say

you're doing or that's where a lot of my

clients get in trouble during reviews

because they say they're doing one thing

but they're they're not doing it uh or

they or they don't they're doing

something that that doesn't have a clear

policy behind it and that it ends up

being kind of a case- by case basis

thing and that's where the a lot of

other clients get into trouble too so if

you are pushing the envelope make sure

you've documented well what you're doing

what its intent is and how you're

administering that and you'll be more

likely to be successful you might still

get a fight on your hands if they don't

agree but if you have the data like Carl

says it's usually

okay yeah um I just want to I actually

100% understand and appreciate Liberty's

perspective here especially because um

accrediting body should be looking to

help push our field forward not hinder

them in a way that makes it difficult

for Innovation which I think and I don't

mean to speak for Liberty but I think is

what Liberty is trying to say right so

an organization like Microsoft who might

not see the return on investment but is

clearly being their certification

program is being led by someone who is

very well respected in the field so she

is following standards and adhering to

probably much higher standards than a

lot of other organizations are but she

has not seen the value that the

accrediting boards are providing her

because of a lack of innovation or the

fact that she's innovating and they

haven't caught up with that yet versus

what should be happening is the

organizations who are not adhering to a

high level of Standards on their own you

should be working with a psychometric

vendor or you should have a

psychometrician on staff who is adhering

to those standards that I mentioned

earlier which are reflected in the ISO

standards or NCCA standards but you can

have a quality program and not be

accredited just like Microsoft is but

what we would love for the accrediting

bodies to do is say hey this is the

standard and I'm open to hearing about

new and different ways to kind of push

the bounds of that and move it forward

and so I think there's like different it

depends where you are on an

organizational level but our industry in

general is very slow to change remote

proctoring is a perfect example of that

like remote proctoring has been around

for a very long time but the only reason

that we were able to kind of

wholeheartedly move in that direction is

because of covid and there was no other

choice but it's been there and it's been

sure technology has evolved over the

time but like it took the ISO standards

to have to go through covid to be able

to get us there right like those are the

kinds of things that we battle with when

we're trying to like move the industry

forward and determining what is

a good um balance of our time and kind

of progressing so I see the value but I

also understand why people choose not to

move in that direction it only takes a

pandemic as Liberty

stated just a little thing like that um

so I'm curious to ask the audience um if

you want to use because I was just

looking in we don't have a any poll set

up I'm just curious using the reactions

below you've got the thumbs up who all

here is is

accredited oh I see a couple

thumbs okay so Rebecca um and feel free

to say no um anything you want to add

regarding being accredited do

you and

okay does anyone out there have any

questions for our panelists regarding

accreditation I'm looking in the chat if

you want to put something in the

chat okay anyone have uh from the panel

have anything that they want to wrap up

with

so I know that I um so for Microsoft

accreditation is probably not something

that we're probably going to pursue but

I will say that the one there was I did

get something out of it right because

the standards are very rigorous and like

I said I inherited a mess when I came to

Microsoft and so if you really like if

if you are looking for a way to improve

the quality of your program I would look

at the standards and see what you can

Implement and maybe and you can

potentially use those as leverage even

if you're not thinking about

accreditation because those standards

are based on industry best practices on

what it what makes a quality program

there's nothing wrong with the standards

in and of themselves so if you need

something as a guide to Get You On Track

if there's particular things that you're

concerned about with your program I do

think it there is value in maybe

leveraging those standards to the it

that might help you so don't get me

wrong in saying that like I don't I I

don't think they're horrible I think

they're there and they do a really good

job and if nothing else for Microsoft it

set us on the right path to where my

what my program is today um but it just

right now it doesn't make any sense for

us because I have a lot of concerns

about our ability to continue to

innovate with some of those

standards yeah I would Echo that I um

Liberty I think everybody who's running

a program should read the standard and

up for inspiration on be willing to be

challenged uh the only other thought I

would say is um if you are in a sector

where you think there

are a lot of low quality programs out

there competing with your program which

is trying to be more rigorous the

standards are a great way to introduce

this discussion with your clients right

A lot of times it comes down like in our

case just the economics like can I

justify doing this and the biggest

reason you struggle is because our

clients don't recognize what they should

care and so I think in particular there

you see there's times where there's like

this you

know mix of like super high quality

programs with like El Cheo you know a

guy on his cap programs and I think uh

engaging clients when that's the case so

they understand that there's a material

difference could be a pathway to getting

that Upstream Demand right because it's

a lot easier to do any of this if the

client is saying I want it I need it a

lot of times the clients don't even know

it exists or why they should care

I I'll Echo that c that was my thought

exactly just in closing um you know I

get routine messages from different

sales teams in different regions asking

about some new website that popped up

and it's using the word certification

plastered all over it um in some cases

we've done a little more digging and

there's no job task analysis being done

there's no standard setting being done

it's just a bunch of questions they slap

onto a test engine and they're calling

the certification so training the sales

team to educate um the clients the

employers really right um that there's a

big difference between one and the other

and and an accredit having an

accreditation kind of play into that uh

does can be a very big

differentiator um so Clyde would you

mind sending Hannah a link to the

standards and we can go ahead and

include that in the um the meeting

minutes yes all right so if there are no

other

questions then Graham I will um pass it

back over to you thank you all the

panelists everybody here today I think

this was really interesting um and um

really appreciate it so thank you

Kristen thank you panel absolutely uh

really enjoyed today thank you so much

um so as we wrap up obviously as always

we'd like to get your feedback about

today's session um the poll as you know

is anonymous and uh what your feedback

does it helps us determine future

presentations you know this is based

about what you like what you don't like

so please do um take a moment to uh you

know to to uh to use the poll um so um

this is the bit I wanted to kind of keep

you all in suspense about um obviously

it's time to mark your calendars if you

haven't already for the itcc for meeting

um hosted at the SAS headquarters or SAS

however you want to say I like SAS

because I think it sounds more sexier um

that's in Kerry in North Carolina on

November 6 7th uh this year so

registrations for the meeting will be

open on August the 1 so uh open August

the 1 we look forward to seeing you

there so thank you everybody for today

and um enjoy the rest of your day thank

you so

much thank you everyone thank you all

thank you so much thank you