How do we evaluate information? Part 1

General comments and chit-chat, or tell us how we can improve KiasuParents.com

How do we evaluate information? Part 1

Postby 3Boys » Fri Nov 20, 2009 7:16 pm

Disclaimer: If you think you know where this is leading, and don't feel you want to raise your blood pressure, read no further :wink:

In the era of information overload and free-flowing information, one needs to be extremely careful about how we evaluate what we read or hear. As much as it may hurt us to hear it, not all sources of information are equal, nor are all methods of evaluating information. There are good sources and methods, and bad sources and methods. You don't have to be dispassionate about a cause or belief, but it is very important to be dispassionate about evaluating information, and apply that consistently. It is FAR too easy to fall into the trap of only hearing 'what we want to hear' (only the good stuff - remember that radio ad?). This is even more so when we feel strongly about a certain issue, and I have seen even some top distinguished scientists refuse to let go of strongly held beliefs in the face of overwhelming evidence, what more mortals like me.

So, as concerned parents, who wish the best health and education for our children, how do we begin to sift the wheat from the chaff?

Let us use the basis of the practice of medicine as a starting point, since that is one I have some familiarity with.

As a layperson, one would hope and wish that every time we consulted a physician, the diagnosis wrought and treatment prescribed would flow from robust and unbiased research, with the application of the best possible scientific methods. If you do not agree with the sentence preceding this, then read no further........

.....still following? Ok.....

So where does your personal doc/PD derive his information from? The following are a couple of scales used to evaluate quality of information. Don't worry if you don't understand the gobble-dee-gook, but it is instructive to read through just to get a feel of what is deemed important.

http://en.wikipedia.org/wiki/Evidence-based_medicine

US Preventive Services Task Force
Systems to stratify evidence by quality have been developed, such as this one by the U.S. Preventive Services Task Force for ranking evidence about the effectiveness of treatments or screening:

Level I: Evidence obtained from at least one properly designed randomized controlled trial.
Level II-1: Evidence obtained from well-designed controlled trials without randomization.
Level II-2: Evidence obtained from well-designed cohort or case-control analytic studies, preferably from more than one center or research group.
Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.
[edit] National Health Service
The UK National Health Service uses a similar system with categories labeled A, B, C, and D. The above Levels are only appropriate for treatment or interventions; different types of research are required for assessing diagnostic accuracy or natural history and prognosis, and hence different "levels" are required. For example, the Oxford Centre for Evidence-based Medicine suggests levels of evidence (LOE) according to the study designs and critical appraisal of prevention, diagnosis, prognosis, therapy, and harm studies:[9]

Level A: Consistent Randomised Controlled Clinical Trial, cohort study, all or none (see note below), clinical decision rule validated in different populations.
Level B: Consistent Retrospective Cohort, Exploratory Cohort, Ecological Study, Outcomes Research, case-control study; or extrapolations from level A studies.
Level C: Case-series study or extrapolations from level B studies.
Level D: Expert opinion without explicit critical appraisal, or based on physiology, bench research or first principles.


So, hopefully one feels somewhat assured that our docs know what they are doing! In fact, I would expect nothing less.

Quite simply, in medicine, as in The Matrix (Merovingian - I have told you. We are all victims of causality. I drank too much wine, I must take a piss. Cause and effect.), there is cause and effect. Proving or disproving it is not always an easy matter. First, let me illustrate what constitutes 'lower levels' of evidence to establish cause and effect.

1) Hearsay

Often, we hear, "Mr X had this condition, and after he took/did/did not do this and that, he improved/got worse....blah blah." We have all come across this at one time or another. For me, it usually comes in the form of 'stock x is going to run in the next few days, better buy!', and since I am still working in a job, as opposed to being retired on a generous stock portfolio, it is self evident that this types of 'evidence' are best taken with a large pinch of salt! 'nuff said.

2) n=1 experiment, otherwise known as anecdote.

The difference with this type of evidence with the above is that there is an actual case that you know of, perhaps yourself even; "I had this condition, and after I took/did/did not do this and that, I improved/got worse....blah blah.". That kind of personal experience tends to be very powerful, and should not be brushed under the carpet. All the same, one needs to be very careful about putting it in context. There are several biases that occur in this scenario, for example, reporting bias, i.e., only positive cases get reported, negative ones do not, seemingly lending strength to a causal link. I will speak about how to control for this kind of bias later.

3) 'Biological' basis

Some of the most ludicrous claims I have seen have been made on the basis of some of the most seemingly seductive biological linkages. Just go to the stuff on colon cleansing to see. On the face of it, extremely simple and logical. But completely bad science and hazardous to boot. Not all examples are so direct, in fact many are subtle. One of my favourites is in the field of sports performance -->http://www.phitenusa.com/. Here is a countervailing view --> http://www.scienceline.org/2008/10/06/a ... -necklace/. I will refrain from posting my personal opinion but drop me a PM if you'd like to hear it :wink:

So, as a concerned parent, will you allow your doc to medicate/treat your offspring solely on the basis of this types of evidence? Probably with some reluctance, yes? That being the case, and acting logically, we should evaluate all health based advice on the same criteria, should we not?

I'll end this part by restating, not all sources of information are equal, and not all methods of evaluating are equal. Quite simply, some are good and some are plain junk. Sorry if this comes across as intellectual snobbishness or condescension, it is not the intent. But it is a serious issue as it deals with (in the case of health care), life and death, so we'd better be serious about it.

3Boys
KiasuGrandMaster
KiasuGrandMaster
 
Posts: 5014
Joined: Tue Sep 02, 2008 5:28 pm
Total Likes: 220


the info world

Postby BerriMore1 » Wed Nov 25, 2009 10:17 am

I totally agree with you.

With so much info around and info gets travel so fast.... we really do not know what is true and what is false...unless we give it time to test. Even given time to test, contradict results always come out.

Not only we have to evaluate the info, we also have to find out the motive of the info..... e.g., there is study saying palm oil is bad for health. Then recent studies say Palm Oil is good for health.

Unless we are the biologist, or we will just have to rely on all the reports and studies. All the studies could be "right" at this era until some studies prove it "wrong". Even if all the reports are lay infront of me, I wont be able to comprehend everything!

E.g.,the wikipedia..., it can be edited by anyone so I am not so sure all the info is correct in there.

BerriMore1
YellowBelt
YellowBelt
 
Posts: 15
Joined: Mon Oct 19, 2009 2:17 pm
Total Likes: 0


Re: the info world

Postby sashimi » Wed Nov 25, 2009 4:39 pm

BerriMore1 wrote:E.g.,the wikipedia..., it can be edited by anyone so I am not so sure all the info is correct in there.


In an era of information overload, official sources are also no longer reliable. People who claim to be official sources or experts in a subject often become a victim of their own certainty, because they don't listen to others or keep themselves updated. How many times have you come across a printed document which is outdated? Or a "professional" in a field who turns out to be hopelessly outdated? Or a newspaper article which seems to be hiding something?

Today, increasingly, information cannot be satisfactorily found in neat packages - I believe that is 3Boys' point.

Knowledge today is a synthesis of multiple sources of information, opinion, interpretation and redefinition. The commonly perceived "weakness" of wikipedia can also be its strength. Because there are people constantly updating it from different sources and angles, it can sometimes be the most reliable.

An encyclopedia, report or fact published a few years ago can already be hopelessly outdated the moment it is printed. Champions take its information, update it and publish it on wikipedia platforms.

Note also that wikipedia.com is only one form of the wiki platform. There are many wikis in the internet, and some, like Confluence, are used internally by organizations for documentation, collaboration and more.

As citizens of the 21st century, as users of Web 2.X (which many people still do not understand nor know how to exploit), it is urgently needed for people to realize that knowledge is how YOU as an individual synthesize it from different sources. This is even more urgent because of the problem of information overload.

True knowledge is not in books nor wikis - at the end of the day they only contain words and data. Knowledge is in how you digest and interpret information - and part of the art of learning is in knowing how to distinguish between fact and fiction, and the many levels of truth in between. :) Teach this to your children! :)

sashimi
BlueBelt
BlueBelt
 
Posts: 499
Joined: Thu Nov 13, 2008 11:21 am
Total Likes: 1


Re: the info world

Postby 3Boys » Wed Nov 25, 2009 5:39 pm

sashimi wrote:
Today, increasingly, information cannot be satisfactorily found in neat packages - I believe that is 3Boys' point.

Knowledge today is a synthesis of multiple sources of information, opinion, interpretation and redefinition. The commonly perceived "weakness" of wikipedia can also be its strength. Because there are people constantly updating it from different sources and angles, it can sometimes be the most reliable.


sashimi,
yes and no. The crux of the issue is that any claim needs to be 'testable'. As I put in a different post, L-carnitine does indeed play an important role in metabolism, but does it naturally mean that more L-carnitine results in more metabolism? Perhaps, but that contention needs to be tested, and the means of testing is really what the scientific method is about. In the testing, one need to be clear about outcomes and controlling for biases. Put L-carnitine in a controlled trial with 20 obese subjects on either arm, exercise them for a month and the treated arm, if it works, would show significant weight loss. Its not rocket science, and not an expensive study to run. Given the financial strength of some of these supplements companies, why have they not published a definitive study on this by now, which would shoot the sales through the roof faster than you can say Viagra? I can only suspect that, it doesn't work......

I am not a fan of wikipedia for many types of information. Non critical general narratives on history, geography, sports and the like, yah, ok, but science and medicine, I certainly wouldn't rely on it, nor should healthcare professionals.

3Boys
KiasuGrandMaster
KiasuGrandMaster
 
Posts: 5014
Joined: Tue Sep 02, 2008 5:28 pm
Total Likes: 220


Re: the info world

Postby ChiefKiasu » Wed Nov 25, 2009 11:39 pm

The question of the value of tacit knowledge over explicit knowledge has always been asked, and never properly answered. That's probably because it has only been in the last decade that the Internet has made possible the mind-melding of millions into a shared consciousness that is accessible to everyone that is plugged in. The way by which entire populations have taken to social computing has been nothing short of amazing - the top 5 Internet services most visited by Singapore are social media sites including Facebook and Wikipedia. The phenomenal take up of open source operating systems such as Linux has been stunning - counter-intuitive even, since it would have been unthinkable that geographically diverse strangers can piece together something as complex and effective as an operating system used by millions of people.

In effect, social media is quickly overtaking the traditional sources of explicit knowledge such as journals and encyclopedias as the official reference. People believe what they hear, if it comes from people that they trust, or if it comes in a form which allows them to shape it easily based on their own value/cultural systems. The Internet did not make this happen - it only amplifies our basic human nature. That is the real power of social media - as a platform that influences popular thinking. Politicians have recognized this long ago, but it is only recently that they are taking baby steps to harness it.

So is the knowledge sourced from the crowds better than knowledge that is gleaned from professionals that are focused on studying the problem in-depth? The logical answer would seem to be NO. Yet history has demonstrated time and again that even professionals themselves can make horrible mistakes. The powerful DEC corporation claiming, in the late 70s, that they cannot imagine why consumers would want to have computers sitting on their desktops, Bill Gates claiming that no application would ever need more than 640Kb, Excite not wanting to buy Google at the expensive price of US$1million, etc.

Crowd-sourcing works only if it is self-moderating, and to do so, there is a need for both opposing and supporting views to every issue being discussed. As Bertrand Russell, in his infinite wisdom says, "The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt. " So the point is, debate is a necessary pre-requisite to deeper understanding of the issues - the problem is that it is difficult to debate without emotions getting into the way.

Use the information provided by the Internet with the clear understanding that there is no DEFINITE answer to the question you are asking, because it depends on the contextual basis which may not be apparent. Evaluate both positive and negative views. The truth is somewhere in-between.

ChiefKiasu
Site Admin
 
Posts: 15221
Joined: Mon Sep 03, 2007 9:16 am
Location: Singapore
Total Likes: 319



Re: the info world

Postby 3Boys » Thu Nov 26, 2009 8:51 am

ChiefKiasu wrote:The question of the value of tacit knowledge over explicit knowledge has always been asked, and never properly answered. That's probably because it has only been in the last decade that the Internet has made possible the mind-melding of millions into a shared consciousness that is accessible to everyone that is plugged in. The way by which entire populations have taken to social computing has been nothing short of amazing - the top 5 Internet services most visited by Singapore are social media sites including Facebook and Wikipedia. The phenomenal take up of open source operating systems such as Linux has been stunning - counter-intuitive even, since it would have been unthinkable that geographically diverse strangers can piece together something as complex and effective as an operating system used by millions of people.

In effect, social media is quickly overtaking the traditional sources of explicit knowledge such as journals and encyclopedias as the official reference. People believe what they hear, if it comes from people that they trust, or if it comes in a form which allows them to shape it easily based on their own value/cultural systems. The Internet did not make this happen - it only amplifies our basic human nature. That is the real power of social media - as a platform that influences popular thinking. Politicians have recognized this long ago, but it is only recently that they are taking baby steps to harness it.

So is the knowledge sourced from the crowds better than knowledge that is gleaned from professionals that are focused on studying the problem in-depth? The logical answer would seem to be NO. Yet history has demonstrated time and again that even professionals themselves can make horrible mistakes. The powerful DEC corporation claiming, in the late 70s, that they cannot imagine why consumers would want to have computers sitting on their desktops, Bill Gates claiming that no application would ever need more than 640Kb, Excite not wanting to buy Google at the expensive price of US$1million, etc.

Crowd-sourcing works only if it is self-moderating, and to do so, there is a need for both opposing and supporting views to every issue being discussed. As Bertrand Russell, in his infinite wisdom says, "The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt. " So the point is, debate is a necessary pre-requisite to deeper understanding of the issues - the problem is that it is difficult to debate without emotions getting into the way.

Use the information provided by the Internet with the clear understanding that there is no DEFINITE answer to the question you are asking, because it depends on the contextual basis which may not be apparent. Evaluate both positive and negative views. The truth is somewhere in-between.


Well written chief, touche :wink:.

I do think that we are talking about 2 different types of knowledge and information though. One the one hand, there is the 'trend' spotting/marketing type 'knowledge'. There are certain types of 'knowledge' to which the didactic, in-depth approaches do poorly, i.e. spotting the sub-prime crisis, the advance of crowd-sourcing et al. The reason is that the trends in these are underpinned by some 'soft' sciences like economics and sociology, to which the inputs and outputs, and effects of interventions are not yet fully (or even poorly) understood. As such, any attempt to be overly definitive in a prediction will invariably end up in grief. To a certain extent, these sciences presently resist attempts at reductionism merely because we do not yet know enough about them. We are learning all the time though, and that we have managed to avoid a major global depression (hopefully!) post-crash is some evidence that federal governments and financial institutions have learnt a little from 1929.

On the other hand, there ARE some types of knowledge that are certainly reducible and amenable to an in-depth approach. In building the 787 Dreamliner, I am sure Boeing utilised a hard science/engineering based approach on many fronts to develop the materials, engines, avionics required for a class-leading jet plane that may revolutionise air-travel. Recently they announced a delay to the programme because testing revealed weaknesses in the airframe that required remediation. Thus the didactic in-depth scientific method comes to the fore, they made some assumptions based on best knowledge on materials strength, tested those assumptions, found them wanting, then went back to re-engineer, repeat. Standard, mundane process, to deliver a high-quality, reliable product.

To illustrate the difference between the two types of 'knowledge'; at the end of the day, Boeing may engineer a marvelous jet plane with industry leading fuel-economy and passenger comfort. Tick one for engineering prowess and execution. On the other hand, Boeing may have completely misjudged the market for a 290-seater 8000 nautical mile range jet plane, and may never sell enough to cover cost of development (remember Concorde?). Cross one for trend and market analysis. Two different 'knowledges', two different techniques for analysis, and possibly, 2 different outcomes.

One needs to distinguish what type of knowledge we are seeking and choose the right technique we use the elucidate the truth, depending on the knowledge we seek.

As ever, apologies for the long post.

3Boys
KiasuGrandMaster
KiasuGrandMaster
 
Posts: 5014
Joined: Tue Sep 02, 2008 5:28 pm
Total Likes: 220


Postby BerriMore1 » Sat Dec 05, 2009 10:54 pm

Good posting to all the aboves.

I think the scary part of wrongly trusted the information is in the field of medical and health info. For example, certain reports shown certain food or vitamin consumption is good for certain health.

Then the marketing people started to use these report to promote their products. Those who believe consumed the product. Then after few years down the road another set of studies proved the result otherwise. :!:

BerriMore1
YellowBelt
YellowBelt
 
Posts: 15
Joined: Mon Oct 19, 2009 2:17 pm
Total Likes: 0


Postby Guest » Sat Dec 05, 2009 11:53 pm

Interesting Insights....so what is Part 2?
Guest
 

Postby 3Boys » Tue Dec 08, 2009 5:05 pm

ks2me wrote:Interesting Insights....so what is Part 2?


The unfortunate partner of hubris is nemesis......I have not gotten to that yet......

3Boys
KiasuGrandMaster
KiasuGrandMaster
 
Posts: 5014
Joined: Tue Sep 02, 2008 5:28 pm
Total Likes: 220



Return to Recess Time