0:00
[Music]
content we use is really trying to
figure out
how do we measure the impact of digital
collections
and how do we do that through all kinds
of
emerging and traditional approaches
methods and metrics
[Music]
[Applause]
[Music]
you are listening to wrek atlanta, and this is lost in the stacks the
research library rock and roll radio show
i'm ameet doshi in the virtual studio
with fred rascoe
marlee givens wendy hagenmaier and
amanda pellerin
each week on lost in the stacks we pick
a theme and then use it to create a mix
of music and library talk
whichever you're here for we hope you
dig it
that's right ameet today's show is called
assessment
as craft we're talking about going
beyond clicks and downloads to tell
stories of impact
and the care skill and ethical
orientation that doing assessment and
libraries requires
we spoke with the principal investigator
for the digital content reuse
assessment framework toolkit project
fondly known
as d-craft and by digital content we
mean
digital materials in the GLAM-R
we've talked about GLAM a lot on the
show but GLAMR
takes the look up to the next glamorous
level
it stands for galleries libraries
archives museums
and repositories GLAMR
the project grew out of the community in
the digital library federation's content
reuse working group and is funded by a
grant from the us
institute of museum and library services
our songs today are about
diverse voices measurements and reuse
understanding how we engage with reuse
and transform digital library materials
is a skill and expertise a craft
a d craft so let's start with the craft
by pottery right here on lost in the
stacks that is a lot of craft
this is lost in the stacks and joining
us in our virtual studio
is santi thompson head of digital
research services
an eva digital research endowed library
professor at the university of houston
libraries
santi served as an inaugural digital
library federation futures fellow from
2018 to 2019
and is the principal investigator for
the digital content reuse
assessment framework toolkit or
craft project which is made possible by
a grant from the u.s
institute of museum and library services
so sanji can you start by telling us a
little bit about your role at the
university of houston and sort of how
you became
interested in reuse and assessment
sure and thank you wendy for having me
and for
giving this opportunity for the
deepcraft project to
get out there to the world so as you
mentioned
i am the head of digital research
services at the university of houston
libraries
and in my current role i manage a
department
that focuses on providing research
services
to those using digital tools and methods
to answer some you know tried and true
research types of questions
and also to answer some new questions
that are now possible with the
power of computation and
uh in that department we also offer
digital repository infrastructure
to archive share and preserve outputs
and works of the scholarly record
but my fascination with use and reuse uh
formed several years ago when i had a
different role
at the university of houston libraries i
started there as the metadata
coordinator
which as you might imagine meant that i
supervise the practice of describing
digital objects
that enter into one of our digital
repositories
and while doing that work sort of
coincidentally one day
i stumbled upon um a set of data
from one of the homegrown services we
have
at the at the university of houston in
our digital library
called the digital cart service which
offers
higher resolution images
to a requester and
how it works is that when
a person would like said higher
resolution image
they provide us with some
raw demographic and usage information
and all of that information was compiled
nicely in a spreadsheet that i came
across one day
and with a colleague and was really
curious to see
we actually had information on um
who was using at least the digital cart
service broadly of course not
specific individual information but we
had
some of the information on who might be
using this and for what reasons
and that kind of information is rare to
find
uh i think in in a digital library
digital repository or even arcable
context
and so we the colleague and i
really dove into that data and was we
were able to publish on it
and that really sparked my curiosity
around
who uses these things and why
um and it's that those two broad
um not overly sophisticated questions i
would say that
have like propelled my interest in this
area
for almost a decade now and that's led
to every
my participation in every one of the
projects that i've sort of built
to do the dcraft project
so at this point having done been
so involved in this work what does reuse
mean to you or or how would you sort of
um
define it in this galleries libraries
archives museums repositories
context yeah that is such a
an amazing and challenging question uh
because our project team has actually
debated this very
question and also what is the difference
between using reuse if there is one
we've debated this literally for years
um
the evolution of our thinking has
certainly proven that it's
really challenging to answer i would say
so if you would have asked me this
question like
two years ago i would have said oh i
know exactly what you is i know exactly
what reuse is and what we have found
through
all of the work in the measuring reuse
project in our own internal discussions
um and presenting out to the world and
getting feedback and input
is that it is not easy and and um they
very well may be
in the same spirit uh and and to be able
to distinguish one from another
is not always clear um
generally i would say uh use
tends to fall on
how materials are accessed
by an individual user through the
institution in which the material is
made available and this is often
discussed
as someone clicking viewing or
downloading
an object from a specific repository
whereas at least at first our
understanding of reuse was going beyond
those
clicks views and downloads to tell more
engaging stories of what people were
doing
sometimes outside of the original
location of this material so
once you've downloaded it what happens
after that
and interestingly enough when we went to
the
um to practitioners in the measuring
reviews project
and we've had subsequent conversations
with folks since then
what we found is that many individuals
do gravitate towards this idea that
moving beyond the initial interaction
with an object
seems to be indicative of reuse and
that knowing those additional
experiences are super important to
telling
better and more engaging stories of
impact
but being able to say this for sure is
x and this for sure is y is a really
complicated
and difficult thing to do now more than
ever
but particularly for those participants
in the measuring reuse project
we are speaking with santi thompson and
we will be back with more about
assessing reuse of digital content after
a music set
file this set under z
six 678.85.c68
you just heard out of use by calva
louise
and before that use it by demay
songs about using reusing things that
are
important
today on lost in the stacks we're
talking about how
practitioners can or should assess the
reuse of digital content they steward
we're talking with sandy thompson head
of digital research services
and eva digital research endowed library
professor
at the university of houston libraries
so let's jump into d-craft what is
the digital content reuse assessment
framework
toolkit and what kinds of challenges is
the project team seeking to address
through the project
so d-craft is a product
of the dlf content reuse working group
and it was awarded
and began awarded by the institute for
museum and library services and began in
2019
we have a really exciting and incredibly
talented project team
comprised of seven different individuals
across the united states and canada plus
project consultants who really
supplement and complement the talent we
have
within the project team and bring a
whole set of specialized
knowledge to the project as well as an
advisor group who really informs and
helps us stay grounded in
thinking of things in a sort of diverse
and
multitude of ways so the toolkit
generally can will contain resources
recommended practices
and use cases for what we hope will be a
sustainable method to measuring and
evaluating
reuse of digital assets held by glamour
institutions
and the purpose of the project is really
to facilitate
how to go beyond traditional metrics
and traditional assessment of digital
repositories um like the
typical reported sort of clicks views
and downloads
in monthly reports or annual reports
and so what we're trying to do is
construct a set of deliverables that
will allow people to begin to approach
that beyond part the toolkit will have
a set of core components first will be
the ethical guidelines and
considerations portion of the toolkit
which will
address key principles for responsibly
assessing reuse
and core to that is being mindful
and respectful to indigenous and
underrepresented communities concerns
and ideas
as well as balancing the pursuit of
assessment with
the heavy implications of privacy for
users
in addition to the ethical
considerations we will also have a set
of
recommended practices which will be
aggregated tools and resources
and will also compile existing
strategies for
assessing various facets of digital
object reuse
and those two pieces are the bulk of
what
our current project team and consultants
are working on
at the moment as we continue to do the
work
of d craft of making the toolkit we will
also
eventually get to the point where we're
developing
training materials and opportunities
which will
be a whole host of resources from
sort of embedded tutorials and quick
start guides to
a more formal curriculum that engages
practitioners
can you talk a little bit about with the
ethical considerations and guidelines
for the assessment
of use and reuse of digital content sort
of how
the project team and the consultants how
you you all have landed on sort of the
set
of values so i have not been as hands-on
with
the ethical guidelines as i have been
with the recommended practices
but i do know that the group who has
been working on
that far more intimately really
look to existing data and other
resources to
highlight some of the key components
or to cover some of the content in the
ethical guidelines
so work was informed by an environmental
scan of existing codes of ethics
and guidelines from various professional
organizations
as well as really relying on that data
we collected in the measuring reuse
project
where we learned for example of
practices that
probably were incredibly problematic or
glamor institutions to be engaging in
particularly where
underrepresented and quote-unquote
marginalized communities
really were taken advantage of or were
not treated with
the respect that they deserved to be
quite honest and so we will take all of
that information
and come out with a what is currently a
draft set of recommendations
and certainly some of those
recommendations
make it challenging to give specific
types of
recommendations they also make it really
difficult we found even most recently to
square the actual act of assessment with
being aligned with some core
principles so first and foremost some of
the recommendations
and the items we're identifying in this
document
are really complicated and difficult to
sometimes come to terms with so when we
talk about
trying to assess traditional knowledge
which might be
information sort of kept within a
set of communities that's passed from
generation to generation
not necessarily in modes or
in ways valued by sort of eurocentric
norms to do any kind of assessment in
that environment or in that
context really requires practitioners to
be
really cognizant first of all of the
of that process and to understand the
values and the
relationship between the the communities
that produce that traditional
knowledge and that knowledge itself
assessing for that
really requires some deep understanding
and respect for
the nature of that that knowledge right
and the
and the groups that have nurtured
sustained and cherished it uh for lack
of better words
and so in that context it's really hard
to say like here's exactly what you
should do
right to measure and to also
report out on what differing audiences
may be doing with that kind of
information
and so what we can only do in those
contexts are say like here are the
the really the things you should be
thinking of as you
move through this work it's not to say
that we're starting a conversation
because we certainly aren't
and it's not to say that we have the
best answers because we probably don't
uh but in what in some way we are i
think
helping to say here are some areas that
a whole host of people might want to
consider
on the horizon moving forward and to do
this in any sophisticated way across
differing categories of knowledge and
different categories of digital objects
we may need to have more voices
and more attention paid to this moving
forward
it may also be that we as a project team
can't
or probably shouldn't try to quote
unquote solve this either
but it might be enough for us to say we
agree that this is challenging
and we hope that collectively
the the sort of powers uh that
make the glamour world uh an amazing
sort of world gen habit can help propel
this and move this forward
it's definitely startling i think to say
here's
a potential set of tools and resources
you can use
to help assess various aspects of use
and reuse
but to know that those tools and
resources bring with them
some problematic interactions with users
particularly around privacy
and we don't have an answer for that yet
and i'm not sure we ever will
you know if a particular tool is owned
by a third-party operator
and they have had maybe not the best
relationship with user privacy that is a
hard
reality to debate and to not necessarily
endorse but at least say
as much as we can this is a tool you
might be able to use
and it might then conflict with these
ethical guidelines we're also coming up
with
as unsatisfying as this might be for a
listener we don't
necessarily have a solution to that
either
but we are aware um and sometimes that's
important
and i think as we do this work over time
and as others
interrogate and also participate in this
work
we can find maybe solutions that both
help you assess
and align with some better ethical
standards and practices
you are listening to lost in the stacks
and we'll talk more with santi thompson
on the left side of the hour
this is michelle casto from the dc punk
archive at dc public library
you're listening to lost in the stacks
on wrek atlanta
[Applause]
[Music]
today's show is called assessment as
craft
as our guest today mentions one of the
components of the d-craft
toolkit will be a set of ethical
considerations and guidelines for the
assessment of use
and reuse of digital content the
guidelines
which have been released in draft
discuss
inclusion diversity equity accessibility
social justice privacy traditional
knowledge
and transparency among other values that
come into play when doing assessment
we'd like to share a few excerpts quote
practitioners should be empowered to
fund
and develop technologies that gather
necessary data
while providing better privacy
protections
glamour institutions are technology
customers
and can leverage their collective power
in the marketplace
to influence vendors to implement
changes on the topic of privacy and
other areas
right on major systems of oppression
work
collectively examples of these systems
include
patriarchy white supremacy racism sexism
ableism
economic hierarchies precipitated by
capitalism and many more
as protection of underrepresented
populations is paramount
this includes not only awareness of
professional obligations
but also the responsibility to push
these boundaries when
the accepted professional codes of
ethics values
or other governing documents do not go
far enough
wow where were you in 2010
file this set under qa qa465.p55
[Music]
that was counting backwards by throwing
muses
and before that we heard measurement by
somersault
songs about different ways of taking
measurements
today on lost in the stacks we're
talking about assessing
how patrons reuse digital content in
libraries archives museums and
repositories
and our guest is santi thompson of the
university of houston libraries
so we've talked a little bit about this
but i guess just to
sort of recap how do you hope that the
toolkit will be used
by different stakeholders yeah how do
you hope that it'll be applied
so i think first and foremost the goal
of the toolkit has always been to allow
primarily practitioners and that's the
focus of our toolkit
to tell stories of impact that
we haven't been able or we haven't been
prepared for
or we haven't had the resources in place
in order to
uh to do my hope uh with the toolkit
is that we can actually
begin to think about nuances in our
audiences
and maybe deeper needs that
individual community members or
different communities may have
as they engage with digital repositories
typically what we focus on for better
for worse is
you know how many things have been
downloaded or viewed
but if we can move beyond that i think
we can begin to have a far
more set of engaging conversations
around
are we actually meeting the needs of
differing communities
and um if we're not how might we be able
to engage them in
in various ways that sometimes manifest
themselves in
the very work that glamour institutions
do um so i don't think it's much of a
stretch to say
if we had better ways to report out on
and then
which individuals were engaging in the
repository
that that would impact how we market the
repository to various audiences
or it might impact how we select for
digitization or acquisition of born
digital materials
of course listeners can feel free to
write me and disagree
if they uh they feel so but i think many
institutions currently just don't have
the data or the
information beyond anecdotes to move in
those directions
to be able to say clearly we should
change our metadata practices
to better engage you know students
uh and and helping them find the
materials that we know we provide for
them
i just don't think we're there yet and
it's sort of my
great hope that something like dcraft
can begin to do that
not only for repositories in an academic
environment but across
all kinds of glamour segments and and
constituencies
i hope that what we've done is created
a resource that is accessible enough in
all kinds of ways
to help those who have never done any
kind of assessment
or who never thought or seen themselves
in the in the
role of assessing digital content i hope
we can
make it so that they really do have a
resource they can turn to
to make this approach not as daunting
and to do that
i mean we have done a lot of work to
really identify alternative tools and
methods
but i think we'll have still some more
work to do to help connect
those disparate sometimes certainly
disparate but also like
tools that aren't necessarily designed
for this but we're sort of
forcing them to do this how to take all
of that information
and make it though a practitioner can
start from
almost nothing and end up with a
compelling story
and not only a compelling story but
maybe multiple stores for multiple
audiences
you need to talk a little bit about how
the events of 2020
2021 um have impacted the project team's
work or maybe
shaped priorities and how do you see
some of these events maybe influencing
assessment and glamours in the future
and sort of
intention around assessment
so i mean certainly physical and psychic
and cultural violence and violation upon
a whole host of
communities is not new and
we had always understood the need for
and had the commitment to ensure there
were diverse
perspectives and voices incorporated
throughout the toolkit
we were pretty clear about that from the
very beginning
including in our application proposal to
imls
so i think if anything the
violence and murder of of folks and the
injustice that we
that we see on a regular basis has only
reinforced
to this group um the need to stay
true to that and to center whenever
possible
diverse voices and ideas in the
construction
of the toolkit and the resources that
are supporting it
i would say in addition to just doing
the work
the core of this group of the project
team and the consultants
some of them we've been together for a
very long time
we've been doing some of us have been
doing this work since 2014
as a defined group and so in that time
you really learn to
appreciate one another to value the
friendship and the
intellect and the curiosity that these
individuals bring
to the work and so part of this
is also about how do we care for one
another and also
sustain and help affirm everyone's worth
and dignity as we do the hard
sometimes really difficult work of
building this toolkit what i would also
say
is that super critical uh over the last
two years has been how do we
see one another acknowledge one another
and make sure that
we are in the best positions and the
best spaces
places mentally emotionally physically
to
carry on this work and we've taken
moments throughout the
the project to really check in with one
another and to be
um supportive of one another and so it
probably shouldn't surprise you then
uh to know that like all of that takes
time right
and that would have prolonged a project
in itself but adding a incredibly
forceful pandemic on top of that uh
has made things even slower so we for
sure
are behind our timeline and we're really
fortunate that
folks like imls and um glf
and our own home institutions are very
supportive and
an understanding of all of that and so i
think
ultimately what it has done is it has
made us more resolute
and knowing that these are priorities
that
diverse voices and perspectives and
prior professional and societal
injustices
are part of this process and and we're
committed to
doing the work to try to make sure that
the toolkit
is reflective of all of those needs
and also reflective of the care
needed to truly sustain a community
because ultimately if if folks are in a
position to
be well enough and secure enough and
respected enough to
do this work it doesn't really matter
what is in the toolkit
it's been hard i would say it's
definitely taken longer
than any of us had anticipated but i
think
at least i hope uh that what will come
from this is a much better product
and a better set of colleagues and
leaders in the in the profession thank
you yeah someone who
is looking forward to learning from the
toolkit i
appreciate that a time is taken to take
care of the people who are
working on it so we've been speaking
today with santi thompson
head of digital research services and
eva digital research
endowed library professor at the
university of houston libraries
thanks so much for joining us santi
thank you for
having me and the project wendy we
appreciate it
[Music]
file this set under h62 dot
[Music]
n663
[Music]
we just heard look like that by sneaks
and each other by public practice
songs about actively listening to
diverse voices
[Music]
today's show is called assessment as
craft
so we may be library practitioners but
we're also users
so what's one piece of digital content
that you've
used or reused over the last few months
fred why don't you start um
well recently i've been taking a
workshop
by a georgia tech professor named brad
rittenhouse he's been a guest on the
show
and we've been learning about uh reading
texts with statistics and analyzing them
with statistical software
are and so i've pulled some uh texts
from hottie trust
and have been playing around with them
uh analyzing
the contents of the text using our
software
you're putting the r in rascoe
[Laughter]
so mine is not nearly as scholarly and
impressive as yours fred
but what came to mind is so my partner
and i
put together this lego set during
quarantine an epic lego set and we made
a video of it like time lapse
and we needed a soundtrack so i of
course i went to the internet archive
and got some public domain music uh as
the soundtrack so
reuse right amanda how about you
um i mean i think i use i
rely and even i rely heavily on the
institutional repository at georgia tech
particularly the uh content that
overlaps with the archives
and i think it's been even more
important to me
in like the virtual working environment
to be able to continue to do like
reference services
and and look at blueprints or techniques
that have been digitized so i'm so
grateful
to um the people that that put that
together and made it searchable
and and make it easier to find things
especially as we're not able to be in
the same place still
shall we roll the credits wendy you want
some public domain music over the
credits
[Music]
perfect
lost in the stacks is a collaboration
between wrek atlanta and the georgia
tech library
written and produced by ameet doshi
amanda pellerin
charlie bennett fred rascoe marlee givens
and wendy hagenmaier
today's show was edited and assembled by
fred
and brought to you in part by the
library collective and their social and
professional network
league of awesome librarians you can
find out more at
thelibrarycollective.org
[Music]
legal counsel and alternatives to google
analytics
were provided by the burrus intellectual
property law group in atlanta georgia
special thanks to santi for being on the
show to the entire d-craft team
and thanks as always to each and every
one of you for listening
[Music]
find us online at lostinthestacks.org
and you can subscribe to our podcast
pretty much anywhere you get your audio
fix
[Music]
next week we continue our covid
restriction schedule with a re-run
and a new show the week after that
it's time for our last song today the
d-craft folks have a hard job
of measuring reuse of digital materials
while also assessing that use ethically
as well as evaluating fairness and
inclusivity of platforms and systems
but no one said that this would be easy
so let's close with
no one said this would be easy by the
postmarks from florida
right here on lost and stacks have a
great weekend everybody