ITCC June Education Session - Panel Discussion on accreditation
Summary
TLDRThe video script features a panel discussion on the topic of accreditation in certification programs. Panelists from various organizations, including Microsoft, the Linux Foundation, and CompTIA, share their experiences and insights on the value and challenges of accreditation, the importance of maintaining standards, and the potential for innovation within the industry. The conversation highlights the balance between adhering to rigorous standards and pushing the envelope to improve and evolve certification practices.
Takeaways
- 📅 The ITCC meeting will be held at SAS headquarters in Cary, North Carolina on November 6-7, with registration opening on August 1.
- 🎉 The panel discussion featured a range of perspectives on accreditation, including representatives from Microsoft, the Linux Foundation, Criterion, Alpine Testing, and CompTIA.
- 🛂 The importance of accreditation lies in providing an independent validation of the certification process, thus lending credibility to the assertion of an individual's professional competence.
- 🚫 The 17024 standard, which accreditation is based on, prohibits a training program from being too closely tied to a certification program to maintain independence and avoid mandating training as a prerequisite for certification.
- 🔄 There is a shift towards more flexibility in accreditation standards, allowing organizations to innovate while still meeting the established criteria.
- 💡 The panel suggested that organizations should be prepared to defend their methodologies and demonstrate how they meet accreditation standards, even if they differ from traditional approaches.
- 📈 Microsoft's experience with accreditation highlighted the rigorous standards and the potential for using these as a guide to improve program quality, even without current accreditation.
- 📊 Data and evidence play a crucial role in demonstrating the effectiveness and validity of new or innovative approaches to certification and exam development.
- 🛑 The cost and effort of maintaining accreditation can be burdensome, and organizations must weigh the benefits against these demands, as seen in Microsoft's decision to not pursue accreditation.
- 🤝 The panel encouraged open communication and willingness to challenge the status quo in accreditation, pushing for innovation while adhering to high standards.
- 🔑 Accurate documentation and clear policies are essential for organizations seeking accreditation or looking to innovate within the certification landscape.
Q & A
What is the main topic of the discussion in the video script?
-The main topic of the discussion is accreditation in the context of certification programs, with a focus on the experiences and perspectives of various organizations such as the Linux Foundation, CompTIA, and Microsoft.
Why is accreditation important for certification bodies?
-Accreditation is important as it provides an independent validation of the certification process, lending additional credibility to the assertion that an individual or organization has met a minimum threshold of professional competence.
What does the 17024 standard refer to in the context of this discussion?
-The 17024 standard, now known as ANAB, refers to the accreditation criteria for organizations that certify personnel. It ensures that the certification process is rigorous, fair, and valid.
Why did the Linux Foundation initially decide against pursuing accreditation?
-The Linux Foundation initially decided against accreditation because their clients were not asking for it and were not familiar with what it entailed, making it hard to justify the process and documentation required for accreditation.
What are some of the challenges organizations face when seeking accreditation?
-Some challenges include conducting a job task analysis with a broad and diverse representation, maintaining thorough documentation, adhering to the standard's requirements, and demonstrating ongoing compliance with the standard.
How does CompTIA benefit from being accredited?
-CompTIA benefits from accreditation by meeting a requirement of certain organizations for doing business, providing internal discipline to their operations, and offering a solid onboarding experience for new staff members.
What was Microsoft's reason for initially pursuing accreditation?
-Microsoft initially pursued accreditation because of a directive from the Department of Defense (DoD) that required accredited certifications to be listed for specific roles in the military, primarily related to security.
Why did Microsoft decide to drop their accreditation?
-Microsoft decided to drop their accreditation due to the lack of return on investment they were seeing and the burdensome annual renewal requirements, which were not justified by the benefits they were receiving.
What is the role of a consultant when an organization is seeking accreditation?
-A consultant can help guide an organization through the accreditation process, ensuring that they meet the necessary standards and documentation requirements. However, it's important that the consultant is willing to push the envelope and support the organization's innovative approaches.
How can an organization demonstrate its commitment to quality without accreditation?
-An organization can demonstrate its commitment to quality by adhering to industry best practices and standards, such as those outlined in the ISO or NCCA standards, and by maintaining rigorous internal processes and documentation.
What is the potential drawback of the accreditation process for innovative organizations?
-The potential drawback is that the accreditation process may stifle innovation by imposing strict rules and standards that can be difficult for organizations to adhere to while also pushing the boundaries of what's acceptable in the industry.
Outlines
📝 Introduction to ITCC Meeting and Agenda
The speaker opens the ITCC meeting by thanking attendees and introducing the format, which will include a discussion on accreditation. The panel consists of experts from various organizations including Microsoft, the Linux Foundation, and CompTIA. The speaker hints at changes and asks attendees to stay tuned for announcements and a poll at the end of the session. The focus is on the importance of accreditation in the certification process and maintaining a separation between certification and training development.
🏛 Importance of Accreditation and Misunderstandings Clarified
Clyde Superat from the Linux Foundation explains the significance of accreditation in certifying individuals' professional competence. He discusses the role of accreditation in providing credibility to certification bodies and the importance of maintaining a separation between certification and training to ensure independence and avoid conflicts of interest. The standard ANSI/ISO/IEC 17024 is mentioned as a benchmark for accreditation, and common misunderstandings about the inability to have both training and certification within the same organization are addressed.
🔄 Linux Foundation's Experience with Accreditation
Clyde shares the Linux Foundation's journey with accreditation, including their initial decision not to pursue it due to a lack of client demand and the costs associated with the process. He outlines the rigorous nature of accreditation, the need for client education, and the internal benefits of maintaining high standards. The discussion also covers the challenges of job task analysis and the necessity of broad community representation.
📈 CompTIA's Perspective on the Value of Accreditation
Carl Bo Bowman from CompTIA discusses the organization's experience with accreditation since 2010, highlighting the benefits of discipline and documentation it brings to their operations. He acknowledges the effort required for annual renewals and the involvement of various teams within the company. Bowman emphasizes the importance of accreditation for revenue opportunities and maintaining quality standards.
💡 Microsoft's Decision on Accreditation and Innovation
Liberty Munson from Microsoft shares the company's experience with accreditation, which was pursued to meet specific military requirements but later dropped due to a lack of return on investment. She discusses the challenges of maintaining accreditation, especially for a small team, and the potential stifling effect of standards on innovation. Munson raises concerns about the compatibility of accreditation standards with new approaches in exam development and accommodations.
🤔 The Role of Consultants and Assessors in Accreditation
The panelists discuss the role of consultants in helping organizations navigate the accreditation process, emphasizing the need to find consultants who are willing to challenge the status quo and support the organization's unique approach. They also touch on the subjectivity of assessors and the importance of having data to back up an organization's methodology and compliance with standards.
🛠 The Challenge of Balancing Standards and Innovation
The conversation continues with a focus on the challenges of adhering to accreditation standards while also innovating in certification programs. Panelists share their insights on the need for organizations to document their processes and stand by their methodologies, even when they differ from traditional approaches. The importance of data in supporting these methodologies is highlighted, as is the potential for accrediting bodies to be more flexible in their interpretation of standards.
📚 Documentation and Governance in Certification Programs
Karina Owens and Zach Owen discuss the importance of having clear documentation and governance processes in place when pushing the envelope with certification programs. They stress the need for organizations to be able to demonstrate their adherence to standards and to have a clear rationale for any innovative approaches they take. The panelists also share advice on dealing with accreditation reviewers and the importance of being prepared to defend one's methods.
🚀 Embracing Accreditation Standards for Program Improvement
Liberty Munson suggests that even without pursuing accreditation, organizations can benefit from reviewing the standards as a guide for improving their certification programs. She emphasizes the value of the standards in setting a rigorous path for program development and quality assurance, regardless of whether an organization chooses to go through the accreditation process.
🎉 Conclusion and Upcoming ITCC Meeting Announcement
The session concludes with a summary of the key points discussed and an announcement of the upcoming ITCC meeting to be held at SAS headquarters in Cary, North Carolina. The panelists encourage attendees to provide feedback through a poll to help shape future presentations. The meeting is set for November 6-7, with registrations opening on August 1, and the organizers look forward to seeing attendees there.
Mindmap
Keywords
💡Accreditation
💡Certification
💡Standard 17024
💡Job Task Analysis
💡Firewall
💡Training Blueprint
💡Exam Development
💡Remote Proctoring
💡Innovation
💡ROI (Return on Investment)
💡Psychometrics
Highlights
Introduction of the ITCC meeting and changes in the format with a focus on suspense and engagement for upcoming announcements.
Excitement about the panel consisting of experts from Microsoft, Linux Foundation, Criterion, Alpine Testing, and CompTIA.
Discussion on the importance of accreditation in certifying individual capabilities and its role in providing credibility.
Clarification on the separation between certification development and training development as per the 17024 standard.
The challenge of maintaining independence in certification processes to avoid conflicts of interest.
Linux Foundation's initial decision against accreditation due to lack of client demand and the costs involved.
CompTIA's experience with accreditation since 2010 and the benefits of discipline and documentation in exam development.
The effort required for annual renewal of accreditation and the involvement of various teams within an organization.
Microsoft's journey with accreditation, the initial push for it, and the eventual decision to drop it due to lack of ROI.
Concerns about accreditation standards stifling innovation in exam development and the challenges of remote proctoring.
The importance of having data to support new methodologies in exam development and the role of evidence in accreditation discussions.
The evolving flexibility of accreditation bodies and the need for organizations to defend their innovative approaches.
Advice for organizations considering accreditation, including the importance of selecting a consultant who supports innovation.
The value of accreditation in differentiating a high-quality certification program from less rigorous ones.
The upcoming ITCC meeting at SAS headquarters and the importance of marking calendars for the event.
Transcripts
thank you for everyone uh for joining
today um we'd like to start out with a
quick statement about uh itcc
meetings um and also um we're we're
going to be switching things up at a
little bit today um so um we're going to
um am I talking too fast or the slides
too slow I'm not quite sure what's going
on SLS are too slow so sorry about that
Graham I can talk slower um anyway so uh
we're going to be switching things up a
little bit today uh so we're going to
keep you in suspense uh so please do
stay tuned uh for
announcements um at the end alongside
the uh the poll that we like to have um
after the
sessions so today do you know what I'm
smiling because it's like uh Hollywood
aists the panel that we've got today so
really excited um our education session
today will include a discussion on uh
accreditation accreditation the panelist
will include Liberty mson from Microsoft
Clyde superat from the Linux Foundation
Zack Owen from Criterion we have also
have Karina Owens from Alpine testing
and Carl Bo Bowman from CompTIA so uh
welcome to you all uh thank you so much
for being here today our a-listers and
um without further Ado I'll pass over to
Christen uh to modate the
panel all right well thank you very much
and um I realized as I was just um
getting ready to go on camera that um I
am wearing one of my favorite its dog
t-shirts so even though it's not cat
shirt Tuesday it's dog shirt Thursday
over here anyway so no I hope no offense
taken to any other venders but sorry
this is a comfy tea so with that let's
go ahead and get started um I wanted to
first start out um for anyone um who is
new to to sort of the idea of
accreditation and whatnot first going to
go ahead and start out and Clyde is
going to talk about kind of the need why
why is there this importance of having
this sort of this this wall between um
certification development training
development um and so on from his
perspective of the of the work he does
outside of Linux foundation so
Clyde let me unmute thanks um Kristen um
so I'll give share some commentary it's
largely informed by uh my rule the past
I think four maybe five years serving on
the accreditation committee for an for
17024 uh which is now called anab not an
um but that's the body that handles
accrediting organizations for certific
person certification which is the 7024
standard um I guess maybe just a word
first on why accreditation at all um and
it really has to do with you know when
we certify individuals through a high
stakes process we are making a assertion
about that person's capabilities right
we're asserting that they are they've
met a met a minimum threshold of
professional
competence uh and the accreditation
process is basically a way of lending
additional credibility to the assertion
that an individual organization makes
right so you can imagine you know an an
org has a CT and somebody that's not
familiar with either the CT or the org
say well why should I trust them like
what what uh evidence do I have that
they've done a good job collecting
defining a good scope what evidence do I
have that theyve done a good job writing
an exam as representative and so on and
so forth and so you know what
accreditation does is provide a level of
independent validation on the process of
yous to come up with the credential that
you created and therefore the the
assertion that you are making about the
competence of the person who passed your
exam and there's a lot of use cases
where there's uh demand for that level
of independent um support for the for
the asss that that you
make uh in particular where most
training organizations or most uh Tech
organizations uh run a ground early is
this idea that that uh can you have
training and
certification both in your org and this
is one of these where there's been a
little bit of
um uh misunderstanding I guess is the
best way to put it uh and there's a
there's a uh belief that taken hold that
that the under the 17024 standard you
cannot have both uh which is not true
what the standard tries to do is to say
uh you cannot have a uh a training
program that is attached to the hip to
your certification program for primarily
two reasons one if your if your training
is basically a rubber stamp version of
the exam
um and you know uh color by members exam
prep what you've essentially done is uh
or pretty much mandated that somebody
take your pay for your training in order
to take the exam because uh it's clear
to the market that this is the approved
way in which that you in which you in
which you can prepare for and pass the
exam and that's not allowed right there
a as a certification
organization uh they don't want uh
organizations uh having a uh a require
what is basically a requirement to take
the training in order to take and pass
the
um the
certification and then the the second
thing they want which is the sort of
flip side of that is uh that there
should be a healthy ecosystem of ways in
which people prepare and it's very
difficult to get a vendor ecosystem
engaged and wanting to prepare people
for an exam if they would on the street
is the organization has made it so that
you basically can't uh compete uh so
what does that mean for organizations is
think about it this way your training or
should be acting in the same way that
they would act if they were a third
party organization and so what that
tends to mean in practice is things like
uh they should be going off the publicly
available blueprint uh that's made
available to everybody right as to
here's what here's the scope and the the
the statement of
expectations uh the what and and by
implication that means they should not
have access to the private exam
blueprint that says okay there's 40
things on here here's the 22 things
we're going to test right and so there
needs to be a clean break between the
the public you know version of the
blueprint that gets publicly made
available and that's what the training
organization can utilize because in that
sense they're no different than any
other third party trainer coming up and
trying to build a training program and
then the second major no no is you can't
have people on the training side of your
house participate in the actual exam
development process for questions uh for
the same reason right so that they you
know because then they get a specific
insight into what
um what exactly is on the exam and how
it's being framed and so the firewall is
ready
uh does the training team have access to
specifics of what's on an actual exam
form or not and uh making making sure
that the people who are writing your
training program were not also actively
developing the content so they can
participate in the GTA along with a
broad Group which which required to have
multiple people providing input on the
development of the scope uh but what
they can't do what you can't do is have
people on your training team then
participate in developing actual items
and go on the exam so hopefully that
creates a little bit more clarity as to
as to what the so-called prohibition on
training is it's a very practically
minded um call to maintain Independence
specifically when it comes to
preparation for the
exam thank you very much for that Clyde
Laura did you have a
question okay so look like audio came up
all right so um the way we are going to
um handle the this call is we've really
got three different groups here right so
we have a Linux Foundation which is
actually looking at accreditation uh we
have CompTIA who is actively
participating in accreditation and then
we have Microsoft who tried
accreditation decided it wasn't for them
and went the other direction um and then
um we will have um Karina and Zach from
the vendor side who can kind of talk
about what it is that they're seeing um
with their customers and so on so that's
kind of the progression that we're going
to go ahead and take um and I've got
some questions for each one of them so
um Clyde let's go ahead and start with
you um you mentioned in the preall that
you guys had Linux Foundation had looked
at doing accreditation before decided no
but you're now back at it um do you want
to tell us
why yeah we
uh the short version is the clients
didn't weren't asking for it and didn't
know what it was and so it was just hard
to justify all the process and
documentation that you have to put in
place in order to be accredited so we
operate our program largely in alignment
with the 17024 standard which is a
standard that you get accredited against
um but there's a there's just there's a
lot of process right ANC uh ISO
accreditation is a process accreditation
right they don't uh they're looking at
are you adhering to the process rather
than your specific
um outcomes uh or specific practices you
it's on your responsibility as a as a CB
is to demonstrate how you adhere to the
requirements of the standard and so you
know we like the rigor of it but we
started talking to the clients and the
clients were just like we don't know
what that is uh and I think for that's
really the business case decision is uh
you know it's nice to have I think it
gives a team a good sense of of
confidence that they're doing things the
right way but there is a that's a cost
right there's a cost to getting the
accreditation there's a cost to
maintaining the accreditation there's
annual
recertification uh checks that you have
to do and then there's a re
accreditation process every think it's
four years uh we do have a major client
that is uh on a new program that's
asking again which is why we're back to
thinking about it but you know
fundamentally it came down to could we
justify the cost of maintaining the
formal accreditation and the answer
before was no and the answer now is okay
if we get this big project then it would
be um a worthwhile investment to
make so um what are as you look at going
ahead and um moving into accreditation
what are some of the initial
accreditation challenges or
processes um that you're looking
at so I think it depends a little bit on
what your starting point is right so
we've operated a process at the LF which
has been
largely intended to be compliant with
the 17024 standard um where I see
organizations most frequently have
difficulty against a standard is uh the
way in which you go about doing your job
task analysis to come up with the scope
of the exam you know what you're
required to do under 17024 is
demonstrate that you've included
feedback from a broadc cross section and
ADV diverse cross-section of your
community your user community and that
often can be a challenge for
organizations that develop their
organizations with a very um targeted
sort of primarily in uh
in-house um expertise right and so
having a going outside to the client
base to the user base making sure you
get representation from in different
industry sectors uh to participate in
the in the job task analysis committee
is often one are the things that
organizations sort of um maybe aren't
doing when they first approach sort of
getting the standard done uh and the
second big one is documentation right
there's a lot of requirement for
documentation throughout the standard
right and so it gets down to the level
of like every document has to have like
a footer that says when last it was
revised and so there's a significant
amount of um administrative capacity
that you have to devote towards staying
in compli lents like do you have CER
statements for um confidentiality for
every single person that's working on
the exam there's a whole checklist of
things um because it's a process
certification that you need to stay on
top of and demonstrate that you met the
standard but also your process for
ensuring that you continue to meet the
standard and so those that that those
are the areas where I think most
organizations when they first come to it
find H those are not things I'm doing
today
uh how am I going to go about making
that happen and how many people am I
going to need you know who's going to do
that that
work great so Carl I'm gonna jump over
to you CompTIA um and my first question
to you um and I apologize if there are
any questions I ask you that you don't
have the answer to because you weren't
involved in that part of it um but how
long has comp have been
accredited sure uh thanks kisten
um we've been accredited I believe since
2010 2011 thereabouts uh that's when we
uh originally uh stepped into this and
it was a significant um undertaking as
as Clyde kind of alluded to uh but we've
been we've maintained our accreditation
since then and I think uh obviously over
the years of maintaining that level of
discipline and documentation I don't
want to say it's become easier but it's
become more of a permanent fixture in
the operation and there's a lot of
expectations around doing things certain
ways to maintain our cord
what value have you seen by um actually
being accredited and staying
accredited um I think there's a couple
things so one and one of the reasons
that we pursued accreditation kind of
alluding to what clad was U mentioning
is we had a a series of different uh
organizations that uh were going to
require uh certification bodies to be
accredited in order to engage in
business and so we saw opportunity for
uh basically Revenue opportunity
essentially and so that was one of the
main drivers there and I think that you
know over the years that's something
that's continuously Revisited um so
that's I think really the impetus behind
it if you will from my perspective
though managing the team that that
develops and maintains our certification
exams I think the other benefit
internally is that it really does bring
a lot of uh discipline to everything
that we do um across the board and
having to document things yes certainly
represents additional work uh and even
some potentially some additional
headcount depending on what you're
trying to accomplish uh in the end of
the day being able to refer back to that
documentation having the documentation
readily available to onboard new staff
members provides a really smooth
onboarding experience and training um it
really is frankly I would argue a good
discipline to have despite whether
you're accredited or not um if you
really want to run run a solid
highquality operation where there
there's a lot of focus in on every
single piece of the operation um it's
definitely a a high standard to work
towards um so I think that's kind of
like a dual answer one is the
organization uh you know kind of Revenue
opportunity and the other is just the
internal I think uh quality standards
that it sets for teams that have to you
know maintain it okay and then how much
effort goes into the um annual renewal
renewal requirements
it's it's a lift uh we have one staff
member I would say
dedicates uh that is really kind of
focused in this area and maintains all
her documentation in the in the
management system and when that time of
year comes she spends easily goes into
like this ansy mode and at least
part-time um you know 50% of our time or
more is dedicated to that and it's it's
at least probably two months or more of
prep uh and So that obviously she man is
uh within our team everything that's
required as well as you know HR finance
and a couple other teams that are also
touched by the standard so um yeah it's
it's a it's a lift I'm not gonna not
gonna
lie okay do you have any lessons learn
or words of wisdom that you would pass
on to others that are maybe like the
Clyde just getting started with Linux
Foundation or or someone else um yeah I
mean you
know it's certainly going to require you
to rethink a lot of things that maybe
you've done a certain way for a long
time and you're like well we're going to
have to do this a different way moving
forward in order to meet the standard or
you might look at it and go we're going
to have to bring on a consultant
part-time to help with this uh every
year to kind of you know help us get
through this this additional lift so
that it doesn't burden the rest of our
team or um the other piece is really
kind of educating other internal staff
members that aren't part of the
development team that might again you
know uh there C pieces that might impact
HR or finance and and spending the time
to actually sit on with those executive
leaders and say look you know we're
going to do this we need you to be on
board with this we need your support we
need you to give us the time and uh and
resources from your team members to be
able to you know check all the boxes if
that makes sense and again willingness
to change right uh you really have to be
willing to think differently and think
from a different
perspective sounds good thank you um
Liberty
um so from a Microsoft perspective you
all went through the process to get
accredited you want to tell us kind of
do I even haveing Clyde Carl talk about
how much work this is so we actually did
it because uh I actually was hired into
micro this is how long ago this has been
I was actually hired into Microsoft to
help us get accredited um and the uh I
was hired at Microsoft in 2006 and we
didn't get accredited until 2008 so
that's just how much work it was now I
came into a mess at Microsoft um we
didn't have anything documented so I had
to create all that
documentation um and but the reason we
did it was because at the time the dod
had a directive 55 55,000 something and
one it was some directive where only
accredited certifications could be
listed and it was needed to uh those
certifications were needed for very
specific roles in the military if I
remember correctly it was primarily
related to security and we wanted our uh
certifications uh a few of our
certifications to be listed in in that
directive so that it could they could be
used to help um the military people in
those roles could use Microsoft right so
we saw it as a potential Revenue 8570
thank you um I've just thrown numbers
out at this point uh so that's why we
did it and so I we didn't it didn't keep
it very long um because we did not see
any return on investment and we weren't
seeing that many more people in the
military choosing a mic to take a
Microsoft certification because it was
part of that directive and the annual
requirement for Renewal was extremely
burdensome now let we also have to
remember that even at that time
Microsoft's exam development team was
pretty small I don't it may have been 15
people but there's always only been one
of me and that was was it was a lot of
work um when I have to manage the entire
psychometric elements of all of our
content um plus then add-on all the
requirements to go around with this
annual renewal so I think uh so we ended
up dropping it because we didn't see any
return on investment but we have been
asked to revisit it even in the recent
past and right now and now the challenge
is more related to the standards not
allowing for Innovation right so if
there are things where I want to
experiment with especially around some
of the stuff that I'm doing related to
accommodations and accessibility I have
a hard time believing that we wouldn't
be challenged for some of those
decisions under the the strict rule of
what those standards say I also don't
think our C renewal um process would fly
and so the reason we have been resistant
on even revisiting this is that in my
opinion these standards stifle our
ability to innovate in places we need to
innovate interesting
so I'd be curious to know and I don't
know uh if anyone here knows but from a
generative AI perspective because that
seems to be something that comes up on
every single one of our calls lately um
how would ancy look at the use of AI in
exam development at
all we talk about stifling Innovation so
I can speak to that a little bit Kristen
um so the 17024 standard is under review
right now right so the standard itself
gets reviewed every eight years or
whatever uh and they they are grappling
with that issue of um what do you have
to say about how and when it's
appropriate to use and it's possible
they might end up saying look at the end
of the day the organization signs off on
the standards and the processes and so
how much do we really care about how
that content came to be so long as you
have the panel of experts with
um experience across industry across
geography you know exp you know
experience levels Etc um and can you
still meet the standard in terms of the
rigor with which they were developed so
there's a whole debate about kind of you
know we've always said it was a process
certification and we didn't really care
about the methods so long as you could
demonstrate that the method meets the
standard and so there might be some of
that happening um
and the same is true at the Liberties
point I think the organization has
gotten a lot better in the past three or
four years about being flexible um with
the fact that there could be other ways
to meet the standard um Co you
know before Co it was a massive lift to
try to get them to approve remote
proctoring because that was one of the
things that was unknown know right it
was like the the whole standard was
written with the assumption that you
would Proctor in person and I think
having to you know every program had to
go to to Virtual and and they had to
sort of pivot ever since then I think
the assessors have been more open to
things that would have been taboo um in
the past and so I guess um to your point
I I would actually encourage
organizations to stick to their guns be
willing to explain how the way you're
doing it meet the standard even if it's
not what they typically would have
accepted which is you know you have to
be you know recertified on an exam or
you have to do a a checklist of things
to demonstrate continue I think there's
more openness now to organizations being
able to articulate more forcefully this
is why we believe this meets the
standard and not have it be met with
this idea of nope that's not how you do
it you have to
do
X I think that's a good call out though
because what really should be happening
um I think so we have two crediting
bodies right so aab is ACC crediting
against the ISO standards and then
there's also NCCA who tends to be a
little bit more prescriptive than um the
ISO standards are for sure um but I
think to Liberty's point is it's a what
should be happening is exactly what
Clyde is saying when you present your
application as a psychometrician as the
organization you should be able to
defend the methodology that you have
chosen and they should be able to say
does the evidence that they've collected
and they've provided me adhere to this
standard I think what happens sometimes
and can create some hesitancy is you you
get assigned an assessor and so that
assessor makes the decision as to
whether or not you are adhering to those
standards and so as Liberty and Carl and
Clyde have all kind of indicated it is a
very involved process that doesn't just
adhere to how are you conducting your
psychometrics but there is a whole
another layer of documentation and
things that you have to put in place and
I think organizations that I've worked
with in the past and they kind of run
the gamut so some are accredited some
have some exams accredited but not their
entire Suite of program
or exams excuse me and others have just
chosen not to go that route or have gone
that route and then decided not to and a
lot of the reason that people get a
little bit nervous about going back into
it or moving in that direction is what
if I get an assessor who doesn't agree
with my arguments and I put in all of
this work to get there and then they
tell me no um and the costs so
everything that's kind of involved in
that but I would wholeheartedly agree
with Clyde that we need more
organizations pushing and saying this is
the standard this is is how I have
interpreted the standard and this is why
I feel like I can move forward and are
adhering not only to the iso standard or
the NCCA standard but I've built my
program around the standards for
educational and psychological testing
and here's the evidence to show that it
makes sense to move in this direction I
just want to Echo that uh what Karina
said is data speaks uh and I think the
proof is in the data as well um and
we've had without disclosing anything
confidential here um several situations
where we've had a bit of an argument in
back and forth and said look we have the
data you can't tell us we're wrong the
data is based on all the psychometric
industry best standards it it says that
everything we're doing is valid the
standard doesn't say otherwise it meets
the standard it is what it is whether
the assessor likes it or not um and
we've had those conversations and I
think Clyde to your point we've seen
that as well that
um you know there there's a bit of a I
don't want to say bias but maybe
perceived bias uh for remote proctoring
and when we show data that says our test
center operation and remot proactor
operation are equivalent and we have all
of the data and psychometric reports to
prove
that it then it is what it is there's
there's no way to kind of argue that the
flip side of that coin is and again
setting the side of the NC standard in
my mind if we were running our operation
and found that there was a discrepancy
and a difference in the outcomes and the
validity of the outcomes between those
two we we would have a problem
regardless and we would have to face the
facts and deal with that right so I
think the anoor is coming through it
just adds a layer of kind of um
accountability for CompTIA right but in
the end of the day you know if we're
going to be accountable to ourselves we
still have to do the due diligence to to
seek out that proof If you will that
evidence and take a hard look at it
whether there's an ANC auditor sitting
there or
not which is kind of where Microsoft has
landed right is I hold a pretty high
standard around everything that we do
but I don't have to deal with the cost
of both in time and resources to
maintain the
accreditation I just want to go back to
something Karina said uh because I think
it is a challenge a lot of organizations
when they do it for the first time hire
a consultant to help them and the
Consultants feel like success means get
the client accredited so I would if
you're looking at doing that I would say
make sure you interview the consultant
for their willingness to push the
envelope because a lot of them will be
will try to put you on this very sort of
prescriptive path of like I know we can
get it through if you would just do it
this way so don't be you if you're going
to go to the consultant which most
organizations do the first time around
unless you have a liberty in
house really push them right make sure
they're willing to take you where you
want want to go because otherwise their
def their default is get you certifi get
you accredited and I think there's a
little bit of a dynamic there where
sometimes the Consultants are staring
people to do it yesterday's way because
they're confident that that way will be
approved so Zach I want to just come
over to you I uh and see what you have
to add what you're
seeing um just your thoughts here I was
actually going to chime in when Clyde
said that actually talk to your
Consultants before you uh you hire them
so that you know that they're okay and
and for I like to say picking a fight
with the accreditors uh but uh basically
just going over the process that you
want to go through them identifying
points that might run contrary to the
standards but still getting you in the
same direction and the same ballpark as
you want to go uh rather than as Clyde
said doing it yesterday's way uh through
a prescribed path that only meets that
doesn't meet your needs effectively and
I and I and along with Carl and probably
Karina as well have had those
conversations with the accreditation
reviewers that says hey you know this is
what we did here's the documentation
here it is on this page of the technical
report or whatever other documentation
or policy we've provided that says we're
doing it up to standards and you're
trying to prescribe me a different way
of doing that we don't want to do that
way that meets the standards but so does
our way and why are you why are you
doing that
so yeah it definitely comes down to your
willingness to actually have less
aversion to conflict in some of these
cases because I think a lot of
organizations fall into that trap of oh
the the reviewer said we can't do it
that way so we have to change not stick
into your guns and actually showing the
evidence that your process is equally
valid all right so um so then I guess my
next question and and I'll open this up
to to all the panelists um so Liberty
had mentioned part of what she was
concerned about was going ahead
and we we know that from a certification
perspective we have to stay Innovative
if we want to keep people interested in
certification we can't stay where we're
at you know pure multiple choice
questions and and whatever we know we've
got to we've got to push the envelope
right and so what I think I'm hearing um
and feel free to expound what I think
I'm hearing is that have a good
consultant or not but have somebody
where you're not afraid to go in and say
this is why we're doing it here's the
data behind it and so on and you can be
successful um so I'll just open that up
to anyone else who wants to add on that
or so um I'm happy to hear it sounds
like there's more flexibility around the
interpretation of the standards I will
tell you Microsoft was the very first it
certification company that went to
remote proctoring and we would like that
was when we were going to be out right
because at the time there was no way
that was ever going to fly that was
really like there it was very much
against the standard as Clyde alluded to
we are also the very first program
that's really tried to be a more
accessible experience and so one of the
things that I have done is I've looked
at all of the security requirements
we've had in place and I've tried to
reduce i' essentially remove the ones
where I think the risk is really low I
have a hard time believing that that is
going to be acceptable under the
standard some of the things that I some
of the decisions I've made on to eror on
the side of the candidate versus on the
side of security in my attempt to be
more inclusive and accessible so I while
I'm happy to hear it as Microsoft keeps
trying to push the boundaries of what's
acceptable in the industry I just have I
I'm gonna just continue to have a hard
time believing that it's worth my time
and effort to go even have a
conversation with the creditors to try
to get us back through that process
but by now you have the data Liberty
that says that the the outcomes in terms
of things like the pass rate and uh
achievement inflation over time were
they materially
different because that's the that's the
argument right it's like look security
isn't for security sake Security is to
protect Integrity of the exam so if
you've done it this way and here's six
months of data that says the average
passing score the time on am the P
values but you know if it's not
different then what do you guarding
against because it's the same
result
I it's a it's a fight worth picking to
cost
point I agree 100% the data speaks for
itself um so I I agree with you Liberty
like there has to be the ROI piece right
if there's no Roi piece I mean
I there's I can't try and convince you
otherwise but um but from
a um an auditor perspective uh when you
have the data all laid out on the
table you can't argue with the data and
we've seen that time and time
again Karina and Zach anything to add
from what you've seen with your
customers who are going through
accreditation and pushing
envelopes uh not so much pushing
envelopes but if you are pushing an
envelope you definitely need to have
what you're doing documented in in
policies procedures your governance
processes and be doing what you say
you're doing or that's where a lot of my
clients get in trouble during reviews
because they say they're doing one thing
but they're they're not doing it uh or
they or they don't they're doing
something that that doesn't have a clear
policy behind it and that it ends up
being kind of a case- by case basis
thing and that's where the a lot of
other clients get into trouble too so if
you are pushing the envelope make sure
you've documented well what you're doing
what its intent is and how you're
administering that and you'll be more
likely to be successful you might still
get a fight on your hands if they don't
agree but if you have the data like Carl
says it's usually
okay yeah um I just want to I actually
100% understand and appreciate Liberty's
perspective here especially because um
accrediting body should be looking to
help push our field forward not hinder
them in a way that makes it difficult
for Innovation which I think and I don't
mean to speak for Liberty but I think is
what Liberty is trying to say right so
an organization like Microsoft who might
not see the return on investment but is
clearly being their certification
program is being led by someone who is
very well respected in the field so she
is following standards and adhering to
probably much higher standards than a
lot of other organizations are but she
has not seen the value that the
accrediting boards are providing her
because of a lack of innovation or the
fact that she's innovating and they
haven't caught up with that yet versus
what should be happening is the
organizations who are not adhering to a
high level of Standards on their own you
should be working with a psychometric
vendor or you should have a
psychometrician on staff who is adhering
to those standards that I mentioned
earlier which are reflected in the ISO
standards or NCCA standards but you can
have a quality program and not be
accredited just like Microsoft is but
what we would love for the accrediting
bodies to do is say hey this is the
standard and I'm open to hearing about
new and different ways to kind of push
the bounds of that and move it forward
and so I think there's like different it
depends where you are on an
organizational level but our industry in
general is very slow to change remote
proctoring is a perfect example of that
like remote proctoring has been around
for a very long time but the only reason
that we were able to kind of
wholeheartedly move in that direction is
because of covid and there was no other
choice but it's been there and it's been
sure technology has evolved over the
time but like it took the ISO standards
to have to go through covid to be able
to get us there right like those are the
kinds of things that we battle with when
we're trying to like move the industry
forward and determining what is
a good um balance of our time and kind
of progressing so I see the value but I
also understand why people choose not to
move in that direction it only takes a
pandemic as Liberty
stated just a little thing like that um
so I'm curious to ask the audience um if
you want to use because I was just
looking in we don't have a any poll set
up I'm just curious using the reactions
below you've got the thumbs up who all
here is is
accredited oh I see a couple
thumbs okay so Rebecca um and feel free
to say no um anything you want to add
regarding being accredited do
you and
okay does anyone out there have any
questions for our panelists regarding
accreditation I'm looking in the chat if
you want to put something in the
chat okay anyone have uh from the panel
have anything that they want to wrap up
with
so I know that I um so for Microsoft
accreditation is probably not something
that we're probably going to pursue but
I will say that the one there was I did
get something out of it right because
the standards are very rigorous and like
I said I inherited a mess when I came to
Microsoft and so if you really like if
if you are looking for a way to improve
the quality of your program I would look
at the standards and see what you can
Implement and maybe and you can
potentially use those as leverage even
if you're not thinking about
accreditation because those standards
are based on industry best practices on
what it what makes a quality program
there's nothing wrong with the standards
in and of themselves so if you need
something as a guide to Get You On Track
if there's particular things that you're
concerned about with your program I do
think it there is value in maybe
leveraging those standards to the it
that might help you so don't get me
wrong in saying that like I don't I I
don't think they're horrible I think
they're there and they do a really good
job and if nothing else for Microsoft it
set us on the right path to where my
what my program is today um but it just
right now it doesn't make any sense for
us because I have a lot of concerns
about our ability to continue to
innovate with some of those
standards yeah I would Echo that I um
Liberty I think everybody who's running
a program should read the standard and
up for inspiration on be willing to be
challenged uh the only other thought I
would say is um if you are in a sector
where you think there
are a lot of low quality programs out
there competing with your program which
is trying to be more rigorous the
standards are a great way to introduce
this discussion with your clients right
A lot of times it comes down like in our
case just the economics like can I
justify doing this and the biggest
reason you struggle is because our
clients don't recognize what they should
care and so I think in particular there
you see there's times where there's like
this you
know mix of like super high quality
programs with like El Cheo you know a
guy on his cap programs and I think uh
engaging clients when that's the case so
they understand that there's a material
difference could be a pathway to getting
that Upstream Demand right because it's
a lot easier to do any of this if the
client is saying I want it I need it a
lot of times the clients don't even know
it exists or why they should care
I I'll Echo that c that was my thought
exactly just in closing um you know I
get routine messages from different
sales teams in different regions asking
about some new website that popped up
and it's using the word certification
plastered all over it um in some cases
we've done a little more digging and
there's no job task analysis being done
there's no standard setting being done
it's just a bunch of questions they slap
onto a test engine and they're calling
the certification so training the sales
team to educate um the clients the
employers really right um that there's a
big difference between one and the other
and and an accredit having an
accreditation kind of play into that uh
does can be a very big
differentiator um so Clyde would you
mind sending Hannah a link to the
standards and we can go ahead and
include that in the um the meeting
minutes yes all right so if there are no
other
questions then Graham I will um pass it
back over to you thank you all the
panelists everybody here today I think
this was really interesting um and um
really appreciate it so thank you
Kristen thank you panel absolutely uh
really enjoyed today thank you so much
um so as we wrap up obviously as always
we'd like to get your feedback about
today's session um the poll as you know
is anonymous and uh what your feedback
does it helps us determine future
presentations you know this is based
about what you like what you don't like
so please do um take a moment to uh you
know to to uh to use the poll um so um
this is the bit I wanted to kind of keep
you all in suspense about um obviously
it's time to mark your calendars if you
haven't already for the itcc for meeting
um hosted at the SAS headquarters or SAS
however you want to say I like SAS
because I think it sounds more sexier um
that's in Kerry in North Carolina on
November 6 7th uh this year so
registrations for the meeting will be
open on August the 1 so uh open August
the 1 we look forward to seeing you
there so thank you everybody for today
and um enjoy the rest of your day thank
you so
much thank you everyone thank you all
thank you so much thank you
Ver Más Videos Relacionados
What is the GSTC?
Pabrik Pengolahan Daging Sapi di Australia Sangat Modern dan Canggih
Understanding the role of a distinguished engineer
Cross-Cultural Communication at an All-Remote Workspace - Internal panel with GitLab team members
Sebuah Produk Impor, Isu Halal Jadi Prioritas (Kimberly-Clark Hybrid Seminar with GroovyEO 204)
Episode 4: Higher Standards (Cambodia)-- Beyond Good Intentions Film Series
5.0 / 5 (0 votes)